Model Robustness Isn't Security

Sven Cattell argues that model robustness, the idea that a classifier stays correct for every input within a small distance of a sample, should not be treated as a security property. He separates the popular meaning of adversarial examples from the mathematical definition practitioners use, then explains how checking every point in that neighborhood becomes unmanageable as input dimensionality grows. Using MNIST as a running example, he estimates how many inputs would need checking even for small perturbation radii, and argues that adversarial training and Lipschitz constraints only partly address the problem.

The middle of the talk covers the trade-offs and real-world bypasses. Robust models can lose substantial accuracy, and for malware classifiers a small false-positive rate matters more than robustness. He describes a production malware model whose false-negative rate spiked months after release because a malware family found a bypass, a problem that robustness would not have prevented. He also walks through older attacks, including spam filters defeated by padding and obfuscation, and an allowlist-style malware detector that was bypassed through its handling of game anti-cheat software.

For the closing takeaways, he recommends basic hygiene first: validating inputs, scanning dependencies, avoiding pickle for untrusted models, hardening pipelines and securing storage such as S3 buckets. He then urges protecting training data as the most valuable asset, layering defenses around the model, monitoring deployed models for drift, modeling realistic attackers, and changing features over time. He also stresses responding quickly when a model fails.