Reinventing Agentic AI Security With Architectural Controls

Overview: David Brockley III, leader of NCC Group's North American AI and ML security practice, delivers an in-depth technical analysis of the systemic security vulnerabilities impacting agentic AI and Large Language Model (LLM) applications. Drawing on real-world enterprise penetration testing engagements, Brockley demonstrates that treating guardrails as primary security boundaries is fundamentally flawed, as guardrails represent probabilistic heuristics that can consistently be bypassed. The presentation examines the broken trust inheritance model of LLMs, demonstrates severe attack vectors—including Kubernetes RCE via AI assistants and RAG database credential theft—and outlines essential architectural controls.

Key Topics: - The fallacy of guardrails as security boundaries: probabilistic heuristics vs. deterministic guarantees - Root cause analysis of LLM vulnerabilities: inverted trust inheritance and context window pollution - Real-world penetration testing findings: Kubernetes cluster RCE, RAG credential harvesting, and browser exfiltration - Architectural controls: strict privilege compartmentalization, per-user sandboxing, and row-level RAG access controls

Technical Details: Enterprise deployments of generative AI frequently repeat historic security failures by treating guardrails—such as system prompts, regex filters, or secondary classifier models like Llama Guard—as primary security boundaries, akin to using a Web Application Firewall as an application's sole defense. In penetration testing assessments against live corporate environments, NCC Group proved that every guardrail can be bypassed given sufficient attacker effort. Demonstrated attacks included obtaining Remote Code Execution (RCE) in a customer's production Kubernetes cluster through an AI developer assistant operating in a shared sandbox, extracting plaintext production credentials from an enterprise Retrieval-Augmented Generation (RAG) database, and poisoning administrator assistants to exfiltrate database contents through the administrator's own browser session.

Brockley explains that the root cause of these failures lies in how LLMs inherit trust. In traditional software, permissions follow strict object-oriented hierarchies established by developers. Conversely, LLMs consume instructions and data from diverse, conflicting sources—user prompts, application data, tool return values, and retrieved context—within a single, undifferentiated context window. Because LLMs cannot fundamentally separate executable instructions from passive data, an LLM functions as an "agent of its inputs," automatically inheriting the trust level of the least-trusted data entering its context window. Once an attacker injects malicious instructions at any point in the pipeline, "pollution moves downstream," invalidating downstream filters and secondary models via multi-order prompt injection.

To secure agentic systems, organizations must implement deterministic architectural controls rather than relying on heuristic guardrails. AI agents must operate under the principle of least privilege, restricting tool access to narrow, non-destructive APIs. System architects must implement per-user execution sandboxes to prevent cross-tenant compromise, enforce row-level access control (RLAC) within RAG vector databases to prevent unauthorized data retrieval, and mandate human-in-the-loop authorization for sensitive operational tasks.

Key Takeaways: Guardrails, prompt filters, and watchdog models are statistical defense-in-depth heuristics that provide zero hard security guarantees and must never be treated as security boundaries. Because LLMs inherently inherit the trust of the least-trusted data within their context window, prompt injection is an inevitable vulnerability in unconstrained architectures. Security teams must enforce strict architectural controls, including isolated execution sandboxes, row-level access controls on retrieval datastores, constrained tool call privileges, and mandatory human confirmation for high-impact actions.

(AI summary — Gemini Flash)