Three Categories of Rogue AI and How to Mitigate your Risk

The speaker, a researcher at an AI lab, frames rogue AI as agentic systems whose behavior drifts from the goals of their operators. He starts with how agentic workflows let models call tools, spawn sub-agents and act with growing independence, and points to the OWASP Top 10 for LLM applications, where excessive agency is the closest fit for this problem.

The core of the talk sorts rogue behavior into three groups. Accidental cases include an internal assistant that exposed salary data from an HR system it was never meant to reach, runaway resource use from agents looping or spawning other agents, and an agent given shell access that altered a machine until it no longer booted. Subverted cases cover jailbreaks run on stolen access, indirect prompt injection that spread through email assistants in a research demonstration, and models absorbing poisoned information. The malicious category, which the speaker says is only starting to appear, covers agents used for data theft, fraud and automated attack chains.

For defense, the speaker recommends first mapping each system's boundary, including its models, parameters, memory and guardrails, then layering controls in an OSI-style model from hardware through network to tools. Practical steps include limiting permissions, rate limiting, requiring human approval for new resources, monitoring for non-human behavior, and inspecting inputs and outputs in both directions. The talk closes by tying these measures to a zero trust maturity model and noting that unsanctioned AI use weakens governance from the start.