Patch Perfect: Harmonizing with LLMs to Find Security Vulns

The talk examines whether large language models help with N-day vulnerability research, focusing on patch diffing. The speakers frame the question as whether LLMs are a real leap forward or mostly sophisticated autocomplete, and they limit the scope to static analysis of decompiled code taken from binary diffs. They begin with an overview of recent academic work, noting that a general-purpose model tended to outperform specialised ones and that giving the model the vulnerability description strongly affected results.

The core of the talk walks through an iterative workflow built against a Citrix advisory, then a Checkpoint gateway bug. Early prompts produced inflated scores, even for changes that only renamed variables. The team improved results by filtering out unrelated components, summarising security-relevant logic before scoring, combining two diffing tools, and running a final ranking pass. Because output varied between runs, they sampled small groups of functions repeatedly and averaged the placements, which they called multi-sample sorting. In the Checkpoint case the fixed function landed in the top ten percent of roughly 300 changed functions.

The closing section covers limitations and takeaways. Context window size and non-determinism remained the main constraints, and a few outlier runs still skewed individual results. The speakers argue that LLMs can give an analyst a better starting point for manual triage rather than replace analysis. They suggest future work such as adding call-tree context, using agents for dynamic testing, and retrieving product documentation to inform the ranking.