I Called Your AI Agent and It Told Me Everything

The talk starts by contrasting text chatbots with voice pipelines, including speech-to-speech models that skip text conversion, and explains why that changes what monitoring can catch. The speaker shows how to tell which kind of system is on the other end of a phone line, and how tone, interruptions and response timing can shift an agent's behavior. A sequence of pre-recorded calls to a simulated bank assistant illustrates the method, which the speaker calls the teapot methodology: start with polite rapport, map the agent's persona and boundaries, test what it refuses, and check what tools it can reach.

The middle section covers the wider attack surface. Caller ID used as an identity check is easy to spoof, DTMF key presses may sit outside the model's view, and retrieval data, user profile fields and conversation history can all carry injected instructions. A live demo shows a profile note overriding an agent's rules. Audio-layer tricks such as sped-up or high-frequency commands are covered, along with error messages that leak internal details and prompts that expose secrets or reasoning tags.

The closing section gives defensive guidance: keep secrets and authorization decisions out of prompts, have tools return redacted data, structure instructions clearly, monitor inputs and outputs, and return generic error messages. The speaker recommends building these negative tests into continuous integration and points to the open-source Voice Goat lab for practice.