Flirting With AI: Pwning Web Sites Through Their AI Chatbot Agents

Paul Brownridge opens by framing website AI chatbots as a new attack surface, with guardrails sitting in front of a language model much as a web application firewall sits in front of an application. He walks through public chatbot failures, from a drive-through order flood to a hiring bot exposed with weak default credentials, and argues that many LLM risks are familiar web flaws under new names. Prompt injection maps to input validation, unbounded consumption to missing rate limits, and excessive agency to over-privileged backends.

The middle of the talk covers case studies and a demonstration. A European rail operator's chatbot validated only the current session, so earlier chat history returned by its API could be manipulated, enabling indirect prompt injection and HTML injection that could lead to cross-site scripting. A March incident in which Morse code decoding was used to move a large sum of cryptocurrency is also discussed. The live demo targets a fictional, deliberately vulnerable legal-firm assistant built on Azure OpenAI with retrieval over blob storage. The speaker probes its boundaries, wraps requests in a legal pretext, asks for base64-encoded instructions, and resubmits prompts to get past refusals, eventually retrieving a confidential lease document through a storage link.

The closing advice is to test AI features regularly, build them with strict input handling from the start, and not assume a deployed model is foolproof. The field is immature, but the speaker argues it is a rich source of findings for assessors.