SADF - A Taxonomy and Evaluation Framework for Agentic Security Failures

The talk argues that many agent security benchmarks overstate success. They count an attack as successful when attack keywords appear anywhere in a model's reply, which also catches refusals that quote those keywords back. The presenter introduces SADF, a framework with an eight-class taxonomy of agentic failures covering tool calls, memory, retrieval pipelines, delegated authority and multi-agent handoffs, paired with a harness that scores only confirmed tool-execution results.

The experiments ran in a simulated environment, with several attempts per attack class. The framework wrapper changed outcomes as much as the model did. With Claude Sonnet held fixed, compromise rates ranged from about 12% under CrewAI to about 31% under SmolAgents. Memory poisoning was the hardest boundary for the Claude models, while delegated authority abuse was rare but potentially high impact. Three attack classes succeeded against every framework tested.

The practical takeaways are to evaluate the orchestration framework as well as the model, give agents bounded identities, filter outputs before they reach other tools or agents, and monitor and review tool calls. High-impact permissions, such as payments, should sit in stricter tiers. The harness and payloads are published on GitHub for teams that want to test their own agents.