Building & Defending a Machine Learning Malware Classifier: Taking 3rd at MLSEC

Andy Applebaum recounts his solo entry into the 2021 Machine Learning Security Evasion Competition, where he built a static detector for Windows executables and then tried to break the models submitted by other entrants. The talk opens with the case for adversarial machine learning, explaining how small, deliberate changes can fool an otherwise accurate classifier, and why the competition imposed strict limits on runtime and false positives.

The middle section covers the build. The detector was trained on public malware and benign corpora, plus a large set of adversarial variants generated with open-source evasion tools and templated payload generation. An off-the-shelf baseline did poorly on the evasive samples, while retraining on the generated variants helped considerably. The final model combined several trained candidates and a stateful check that compared new submissions against recent ones. It finished third.

For the attack half, the speaker queried rival classifiers, trained a surrogate model to mimic them, and used reinforcement-learning evasion tools against it. Reconnaissance and other frameworks were also tried, and many submissions were filtered out or failed to evade. Traditional packing and encryption tools proved far more effective at evading classifiers that had no training on them. The closing lessons cover planning disk and compute resources, keeping careful experiment records, separating training and test data, and the ways contest scoring can be gamed.