LLMs in AppSec: Why They Still Need a Chaperone

The talk opens by contrasting large language models' success in areas like content drafting, translation and data analysis with their weaker fit for application security. AppSec demands exact, reproducible results, clear evidence for sign-off, awareness of a constantly changing threat landscape, and an understanding of how a system behaves at runtime. Mistakes can carry severe consequences, so prompting alone is not enough to meet that bar.

Using concrete examples, the speaker shows how a model reasoning over static inputs such as architecture diagrams and configuration files can miss services that open random ports at runtime, unless live scanner output is fed into it. The talk covers package hallucination, where models suggest dependencies that do not exist and attackers can register them. It also describes an experiment in which a model was asked to write static analysis rules and to build exploits that confirm or rule out findings. The model often produced fake proofs, and it struggled with confident, small incremental edits that repeatedly failed to build.

The practical takeaways are to break the work into small modules, give each component only the tools it needs, state requirements explicitly, and rely on deterministic tools and careful data preparation. Supervisor agents, output templates and frequent quality checks help keep results consistent, and humans should stay involved at chosen decision points. The talk closes by framing the goal as building and testing a working system around the model rather than trusting the model on its own.