I am a weapon of the mind

The talk opens with a common customer question: will large language models change social engineering? The speaker gives a short, non-technical explanation of how these models predict the next token from large amounts of public text, and argues that at scale human writing is quite predictable.

The core of the talk is a phishing experiment run on a security awareness platform. An experienced internal red team wrote four phishing emails, and the same prompts were given to ChatGPT to produce matching versions. Across more than 53,000 simulated sends to users in over 100 countries, the AI-written emails had a failure rate of about 2.9 percent, while the human-written ones reached 4.2 percent. Reporting rates were close for both. The gap narrowed for users with more phishing experience, and the speaker noted that the AI emails read as more formal, whereas the human emails were more aggressive and urgent. Country results varied, with some differences tied to the fact that the emails were in English.

The speaker concludes that language models can already produce lures that work, and that they could make individually tailored attacks cheap. Existing email filtering and awareness training still apply, so defenders should keep those controls current and test these tools themselves.