Malware AV Evasion Tricks: Cryptography In Malware

The talk looks at why common encryption schemes in malware development, such as XOR and RC4, are easy for antivirus engines and analysts to spot, since they are widely reimplemented and recognized. The speaker explores whether less common, textbook ciphers can reduce detection when used to encrypt payloads, strings and command-and-control traffic.

Test samples start from a Metasploit payload, a message box, or a simple reverse shell, then get encrypted with several algorithms, including Z85, TEA, A5/1, Madryga and Skipjack. Each variant is compared on Shannon entropy of the PE sections and on VirusTotal detection counts. Several ciphers cut the score noticeably, for instance from 31 to 17 or 24, while entropy rises. Lab tests against Microsoft Defender, Bitdefender and Kaspersky showed similar bypasses, and Slack and Telegram API channels avoided detection where a custom C2 server did not.

The main takeaway is that a single cipher rarely works on its own. Results held up when several algorithms were chained with base64 or similar encoding, but entropy climbs with every extra round. The speaker also shares proof-of-concept code and YARA rules for these implementations so defenders can detect them, and notes that some rules produced false positives.