Field Notes on Offensive Agents - Reusability, Reliability

The speakers, from a security consultancy, start with the gap between flashy offensive agent demos and systems that can be trusted on client engagements. Their framework runs in four stages: discovery and reconnaissance, planning of vulnerability hypotheses, attack by specialised agents, and validation. Models are pluggable so different agents can use different models, and shared tools such as a browser can be reused across agents. A human still reviews findings before anything reaches a client report.

For reusability, they recommend turning a prompt that works for one vulnerability class into a template, sharing tooling instead of duplicating it, and packaging single agents so consultants can use them in their own harnesses. For reliability, structured output makes final reports easy to parse, validators cut false positives, and retries with backoff stop long runs from crashing when an API or tool fails. Authentication was the hardest problem, so they moved login flows into a dedicated subsystem that tracks sessions, cookies and keys.

On evaluation, they argue teams must define what success means, choose deterministic measures where possible, and decide what pass rate justifies release. They tested against internal labs, then bug bounty targets, to judge new models and prompt changes. Safety measures include an ask-human tool, a proxy that blocks out-of-scope hosts, careful wording in agent setup, and a kill switch. Their takeaway is that agents work best as a force multiplier running alongside human testers.