Why LLMs Fall Short in Vulnerability Management

Itamar Sher, who has spent about fifteen years in vulnerability research, opens by challenging vendor claims that LLMs can take over vulnerability management. He starts with a review of existing research, which suggests models handle short, simple bugs reasonably well but struggle as scope grows, with weak grasp of program flow and logical errors. He then describes four small experiments: identifying CVEs in single-file C code, measuring false positives on patched code, generating and judging security patches, and flagging malicious commits in JavaScript dependencies.

The results were mostly discouraging. Correct weakness classification and correct function identification were both low, and the model often misread the vulnerability entirely. Leading prompt wording pushed false positives from about 45 to 55 percent. An LLM used as a patch judge was wrong about 40 percent of the time, so manual review was still required. The commit scanner caught all twelve real-world malicious commits in its test set, but it also flagged poor coding practices and remained exposed to prompt injection.

The closing advice is to keep prompts neutral and concise, supply relevant context but not unrelated code, run models repeatedly because outputs vary, and keep a human in the loop. The speaker found LLMs most useful for suggesting patches, assisting with backports, and building in-house commit review tooling, rather than trusting them to triage findings unattended.