AgentBreaker A blind spot detector for your coding agents

The talk starts from a shift in software development: coding agents write code far faster than humans can review it, so the bottleneck has moved to PR review. The speakers, from Corridor, argue that security should be enforced when code is generated rather than after vulnerable code is already in a pull request. They cite a benchmark result in which even a strong agent setup reached only a small share of secure solutions, though most outputs were functionally correct.

The core idea is security context retrieval. Before an agent implements a plan, a separate model identifies the security risks that the planned change could introduce, and those risks are added to the plan. On 30 SussVibes tasks, a Claude Code setup with this step raised security correctness from 36% to 52% while functional correctness stayed at 100%. To measure the retrieval step itself, the team built PreMortemBench. Frontier models scored F1 around 60 to 63%, and their latency was too high to run before every plan.

To close that gap, the team fine-tuned an open-weight Qwen model. It was faster and scored higher on their internal benchmark, with most of the gains on topics that few models retrieved correctly. The talk's broader takeaway is to measure agent harnesses on the risk axes you care about, and to weigh latency and precision alongside accuracy. The AgentBreaker framework, which extends AutoBencher to full agent harnesses, is released for that purpose.