Smack My LLM Up!

The talk opens with a few quick demonstrations of fooling AI systems. A steganography-style overlay caused a computer vision model to misdetect an image, and a chat model was reportedly coaxed into giving dangerous chemistry guidance through a framing trick. The speaker then moves to the main case study: a coordinated scheme that targets social media influencers, mostly women, by mass-reporting their accounts until the platform locks them out.

Victims are approached by fake support contacts who demand payment to restore access. After payment, the attackers obtain email and social account credentials, then trigger an automated support workflow that unlocks the account through a crafted reply. The speaker describes contacting the operators, observing their coordination on messaging apps, and analyzing a victim intake web page that appeared poorly secured and held victim data. Natural language processing was used to test which phrasing the support model responded to, and the speaker shares that analysis as a Jupyter notebook.

The takeaway is that automated support pipelines and LLM chatbots can be manipulated by crafted input, and that threat hunting benefits from applying NLP to scam and phishing text to link campaigns and actors. The talk also flags the ethical questions that arise when researchers probe criminal infrastructure and victims directly.