Exploiting Emergent Property-Based Vulnerabilities in Large Language Models

The talk begins with the idea that emergent behavior appears when complex systems interact, and that it is hard to predict without scaling them up. The speaker applies this to large language models, which are usually wrapped in a middle layer of filters, retrieval components and prompt assembly. Attacks that get past that layer can reach the core model more directly.

Several techniques are demonstrated. One anchors the model to an earlier year so that its guardrails respond differently. Another stacks nested fictional contexts until the model loses track of its constraints. A third uses poetic phrasing in a non-English prompt to coax out content it should withhold. The speaker reports that some of these worked across many commercial and open models. He also covers a roleplay attack against a voice-driven game character that exploited contradictory instructions and a small context window, and persistent tuning keywords that altered chat behavior.

Disclosure to AI vendors was often unproductive, according to the speaker, who argues that the middle layer should be transparent and auditable. The main takeaway is to test these systems hands-on, observe how responses shift with wording, and document the results, since mapping these behaviors is work no one can do alone.