Hidden Risks of Integrating AI: Extracting Private Data with Real-World Exploits

The talk starts from a report stating that 13% of organizations had breaches involving their AI models or applications. The speaker argues that adding AI widens the attack surface in ways standard application security thinking does not cover, and that language models are probabilistic, so they cannot serve as a reliable security control. A short neural network demonstration shows how models learn from data and why their accuracy stops short of perfect.

The middle section focuses on extracting private data. A Llama model fine-tuned on synthetic personal records refused direct questions, but repeated queries eventually exposed a full record, including a passport number. Retrieval-augmented generation spreads copies of sensitive data into vector stores, prompts and logs. The speaker then showed that embeddings can be partly inverted with an inversion model, recovering names, dates and dollar amounts from a short message. Tool calls through model context protocol servers were also shown to leak an SSH key and note contents, driven by a malicious tool and hidden text in a note. Publicized incidents involving Microsoft Copilot and a calendar automation feature were cited as real-world examples.

The closing takeaways are to assume failure and guard against consequences, to mitigate data risks because AI systems multiply copies that are rarely monitored, and to trace every untrusted input to any action it can trigger. Practical steps include limiting automatic AI features, holding vendors to account on security, and encrypting data before storage, including vector encryption that resists inversion.