Security Chats: Defending Agentic AI with Deterministic Guardrails

### Overview In this installment of Microsoft Security Response Center's *Security Chats*, host Joe Miller speaks with Santiago Zanella, a principal researcher in AI security and privacy at Azure Research. The discussion centers on the security challenges emerging from increasingly autonomous and agentic AI systems, the inherent limitations of probabilistic defenses (such as secondary classifier models), and ongoing research into deterministic guardrails, specifically using information flow control and the Dual LLM architecture.

### Key Topics - **Transition to Agentic Autonomy:** As LLMs gain agency, tool execution capabilities, and browser or computer-use interfaces, they are entrusted with confidential data and credentials. Operating without robust controls risks severe prompt injection and data exfiltration issues. - **Offense vs. Defense Balance:** Santiago highlights that utility currently outpaces security across the industry, giving attackers an advantage. However, AI may eventually provide defenders with a net security gain in the long term through proactive attack surface exploration. - **Failure of Probabilistic Mitigations:** Relying solely on probabilistic AI classifiers or layered model prompts provides only a first-line filter against unsophisticated attacks. Motivated attackers consistently bypass these soft layers, necessitating deterministic, traditional software engineering boundaries.

### Technical Details - **Dual LLM Architecture & Information Flow Control:** Because transformer architectures inherently blend instructions and untrusted data into a single channel, full separation inside the model is fundamentally impossible without severely crippling utility. Building upon Simon Willison’s Dual LLM design pattern—which separates a privileged planning LLM from a quarantined LLM handling untrusted data—Santiago's team incorporates information flow control. Data ingested from outside tools is tainted with labels (such as 'untrusted') that propagate across contexts. High-stakes actions (e.g., banking transactions, external emails) are governed by deterministic security policies that prohibit execution when context contains untrusted or tainted data. - **Variable Passing & Performance:** Rather than injecting raw tool outputs into the planner context, variable passing stores tool results in orchestrator memory and passes references. While current multi-LLM architectures see an approximate 3x token usage expansion, Santiago projects optimizations can bring overhead down toward 10%. - **Key Vulnerabilities and Research Tools:** - *Embedding Inversion:* Techniques like 'vec2text' demonstrate that high-dimensional semantic embeddings can be inverted to reconstruct original text verbatim, requiring embeddings to receive the same confidentiality safeguards as raw data. - *Optimization-Based Attacks:* Tools like GCG (greedy coordinate gradient) allow prompt optimization on open-weights models to transfer successfully against proprietary models. - *Security Platforms & Frameworks:* Mentioned testing tools and benchmarks include PyRIT, the UK AI Security Institute's Inspect AI, Gray Swan Arena, Lakera's Gandalf challenges, and MSRC's LLMail Inject competition.

### Takeaways - **Deterministic Harnesses over More AI:** The belief that prompt injection can be cured by 'throwing more AI at the problem' is a critical misconception. Reliable protection requires treating LLMs as stochastic black boxes encapsulated within deterministic harnesses. - **Ecosystem Advice:** Security researchers of all skill levels are encouraged to explore AI security through hands-on CTFs, bug programs (such as Zero Day Quest), and community resources, analyzing the fragile assumptions built into emerging autonomous systems.