Or am I going about this all wrong?
Yup.
Source or destination port will not tell you anything about what the user was using.
You'd want to check the IP and port compared against an archive of the Tor consensus at that time, to see if any of them match known ORPorts.
However the user may have been using bridges, in which case you will not be able to tell from just an IP and port used.
You can retrieve the IPs and Ports for relays in historic consensus from the Collector Microdesc Archive.