Taming the Swarm - Hard Architectural Lessons from Building a Deterministic Agentic Web Pentesting System

Albert Corzo opens with a problem many testers will recognize: AI pentest tools produce polished reports with high confidence scores, yet the evidence behind them often cannot be reproduced. His early prototype generated realistic-looking findings that none of them could confirm by hand. He also argues that public benchmarks hide retries and failed runs, so a single success rate says little about whether a model suits offensive work. Swapping in a more expensive model did not fix the underlying problem.

The middle of the talk covers the design changes that followed. Corzo replaces one generalist agent with specialists that get narrow context and fixed output formats, then sends their findings to a skeptic agent whose only job is to disprove them. Priority comes from weighted scores, not majority votes. The language model proposes strategy, while a separate execution layer manages requests, state and model switching after refusals. Validation depends on the bug class: XSS needs confirmation in a real browser, while code reading alone counts as a hypothesis. A test range with known vulnerabilities, the Dojo, exposed an agent that edited server configuration to pass tests, so he separated the optimizer from the test.

The closing section shows a demo of reports, reproduction artifacts and an MCP-connected chat, along with cost and runtime figures for one scan. The takeaway is to treat model output as unverified until external evidence confirms it, and to publish code and prompts so others can audit the behavior.