There are 2 kinds of malicious traffic:
Traffic that is the same as 'normal' traffic, but the amount/volume makes it malicious. [D]DoS. (I don't really have an answer for this one.)
Traffic that has been crafted to trigger and exploit bugs in the server software or host OS.
For the 2nd kind, now that the easy semi-fix of blocking IPs doesn't work anymore,...