Don't Make This Mistake: Painful Learnings of Applying AI in Security

The talk examines an attempt to use a generative AI model to automatically remediate application security vulnerabilities in source code. The presenters began with a simple prompt template and tested OpenAI models against deliberately vulnerable web applications. The results were weak: only a minority of suggested changes were sound, many edited unrelated code, and some fixes broke application logic or did not compile. A live demo of a reflected cross-site scripting issue and a SQL injection fix showed how the same request could produce very different outcomes.

The middle of the talk covers techniques that improved reliability. Custom prompts tailored to each vulnerability and code pattern, limiting context to the relevant method, and asking the model to prefix each code line so output could be parsed without relying on JSON all helped. The presenters then described a hybrid design, where deterministic syntax-tree transformations apply the fix and the model supplies only small pieces, such as a conditional check. They also noted that rule-based tools from established vendors take a similar route, but writing and maintaining those rules for every pattern is slow.

The takeaway is that AI works best as a supervised assistant rather than a standalone remediation engine. Model output is non-deterministic, so practitioners should expect to verify every change, run regression tests, and keep the model's role narrow. Automated fixes can reduce remediation time, but they do not yet remove the need for review at scale.