Avoiding The Basilisk's Fangs: State-of-the-art In AI LLM Detection

The talk starts from the idea that large language models produce text that drifts toward the average of their training data, so their output tends to be more predictable than human writing. The speaker frames detection around perplexity, the surprise a model would feel at each word, and asks whether that signal can be estimated without running a large neural model.

The proposed tool, Zippy, uses LZMA compression, an older technique used for anomaly detection. A corpus of model-generated text is compressed first, then a candidate sample is appended and the change in compression ratio serves as the score. The speaker compared it with several detectors on a test set of about 5,000 documents. Zippy reached roughly 80% accuracy and 86% area under the ROC curve without tuning, ahead of OpenAI's withdrawn detector at 62% and running in minutes rather than hours. Only one commercial service scored higher. Known weaknesses include short inputs, random noise and text in other languages.

The second half focused on evaluation. Public datasets are sparse, often unprompted, and carry biases such as stiff formal letters. Commercial detectors rarely publish their methods and change over time, which makes results hard to reproduce, and the speaker argued this matters when detectors inform decisions about students. The talk closed with a call for standardized benchmarks, a browser extension that fades out likely AI-written paragraphs, and an invitation to collaborate on the open-source code.