DoH Deception: Evading ML-Based Tunnel Detection with Black-Box Attack Techniques

The talk opens with background on DNS tunnelling, which attackers have used since the 1990s to move data and command traffic through DNS, and on DNS over HTTPS, which hides those queries inside HTTPS. Detection has moved toward machine-learning models, with ensemble methods such as gradient boosting currently leading. These models are trained on flow features extracted from packet captures, and the research tool used throughout is DNS2TCP.

The core of the work is a black-box attack in which the attacker only sees the model's inference output. It uses a zeroth-order optimization method adapted from the ZOO attack. The unconstrained version reached high success but produced impossible values, such as negative durations and enormous packet sizes. To fix this, the speaker added per-feature limits taken from the tunnel tool and a list of features allowed to change. The constrained version succeeded less often, but a single success revealed feature values that could be reproduced by configuring the tunnel tool.

A live demonstration ran a gradient boosting detector against a tunnel on port 443, flagged it as malicious, then adjusted packet size and flow time values until the same traffic was classified as benign. The speaker presented the approach as applicable to other models and suggested it may extend to network intrusion detection systems. Defensive suggestions include adversarial training, avoiding features a user can easily control such as timing, and auxiliary detectors that catch such manipulation.