Trust Amplification in Enterprise AI Systems - MS CoPilot Case Studies

The talk starts from the observation that phishing succeeds by earning trust, and that users are now trained to spot it. Enterprise AI assistants create a new channel that users are told to trust. The speaker introduces "trust amplification": attacker-controlled content gains credibility as an AI system converts, contextualizes and redistributes it.

Three Microsoft Copilot case studies follow. In the first, a Word document carrying a spreadsheet formula injection was converted to Excel by Copilot, and the payload survived into a cell that could run a PowerShell command once the user enabled content. In the second, an attacker who started a shared Copilot session could poison its responses through an injected document, so every guest who joined received a link that led to an arbitrary protocol handler. In the third, a hidden instruction made Copilot walk a user through a device code sign-in as if it were a legitimate feature. A short demo showed the resulting token granting access to files, contacts and sharing, which allowed the lure to spread to other users.

The speaker's defensive advice is to treat all AI input as untrusted, sanitize outputs before conversion, limit how long shared sessions persist, train users on AI-specific social engineering, log AI activity, and hunt for device code authentication in sign-in logs. The talk closes by presenting Locusta, an open-source toolkit for red teams and defenders to model how influence spreads through AI-assisted environments.