AI Pentesting is not a Vibe Check

The speaker, a staff AI security researcher with a web application and bug bounty background, argues that testing AI-enabled products should go beyond the chat window. The focus shifts to every place where outside data enters the model's context and every tool or display path the model can influence. Prompt injection serves as the delivery method, while the resulting impact is usually a familiar application flaw.

Examples come from several products. A merchant assistant on an e-commerce site could be steered into a DNS lookup tool to leak a coupon code, since DNS traffic slips past HTTP-focused controls. Hidden instructions in an image or in white-on-white sticky note text reached a workspace assistant that humans could not see. A tool server's fetch function turned blind request forgery into a partial read, and a tool description carried exfiltration instructions. Model output inserted as raw HTML produced cross-site scripting, and a tool-output sandbox trusted messages from any origin. Two denial-of-service cases involved a model looping through reminder tool calls and a markdown payload that crashed an admin view on every reload.

The main takeaway is that the highest-impact findings lived in the application around the model rather than in the model itself. Testers should map what the model reads, what it can call, how its output is rendered, and which privileges it inherits. Existing skills for SSRF, XSS, access control, and rate-limit testing still apply.