There is no way to currently do this with Tor and it's very unlikely to ever be added.
What you're describing would be "Deep Packet Inspection", recognising protocols by inspecting the data streams in packets.
This is not currently possible on Tor, the only thing it can do it write a warning to the logs that an "unsafe" port number is being used. This can be done by adding a WarnPlaintextPorts
directive to your torrc.
It's also unlikely to ever be added in Tor, performing parsing of adversary controlled data passing through Tor to recognise protocols opens it up to remote attacks, Tor would need to parse an arbitrarily large set of protocols to try and identify which is and isn't "plaintext".
Even if the Tor Project was able to actually get the dev time to write a full DPI toolkit with a large set of protocol parsers and recognisers. This method is still destined to fail in cases where Tor's recogniser fails to determine which protocol it is: should it assume it's plaintext and potentially block communications which are secure? or should it allow it and potentially fail to perform it's duty?
Given that some of Tor's Pluggable Transports are designed to defeat DPI by creating protocols that can't be definitively recognised even with knowledge of what the protocol is, the futility of this kind of approach should be apparent.
Tor need not and should not care about what the data it's proxying is. It must be the responsibility of the user or, better yet, the application itself to ensure that appropriate encryption is properly deployed as required.