Creating the Torment Nexus: Using Machine Learning to Defeat Machine Learning

The talk opens with an analogy to adversarial noise, which can make an image classifier misread a picture, and asks whether machine learning tools used in cyber defense face the same weakness. The speaker focuses on EMBER, an open model trained on features extracted from Windows executables with a feature extractor. The model returns a percentage chance that a file is malicious.

Using SHAP explanations, the speaker finds that the model leans heavily on header fields such as the compile timestamp, debug table size and OS version. Changing these fields does not alter what the program does. Editing a few of them on a known malware sample lowers the score substantially, and a search method called particle swarm optimization, run over seven features, pushes the score close to zero. The speaker reports similar results on about a dozen other samples, and a noticeable drop on VirusTotal as a side effect. The modified file still runs.

Proposed mitigations include removing inputs that carry no meaningful signal and building features from control flow, which describe what a program actually does rather than incidental properties. The takeaway is that AI products in security should be scrutinized before deployment, especially as vendors rush to add AI to everything.