Don't be LLaMe - The basics of attacking LLMs in your Red Team exercises

The talk starts from a gap the speakers see in red team practice: discussions of generative AI often focus on safety and ethics, or stop at the model itself, when the security risk sits in the applications and agents built around it. Brent Harrell gives a short, math-free overview of how large language models work. Neural networks produce probabilistic predictions, text is processed as tokens, and an attention mechanism weighs surrounding context inside a limited window. Because a bare model only generates text, its impact depends on the tools it can call and the data sources it can read.

The second half, led by Alex Bernier, covers attack paths. Prompt injection and jailbreaking are presented as entry points, with techniques such as hiding instructions in invisible Unicode characters, obfuscating requests through other languages or encodings, persona setting, storytelling pretexts, and confusing the model across a conversation. The more useful material concerns impact. Instructions planted in documents read by a retrieval-augmented knowledge base can lead to data exfiltration through rendered markdown images, poisoned answers that steer users toward malicious downloads, or access to content the attacker could not open directly. Real engagement examples include a helpdesk bot that performed password and MFA device resets, and a code-generation tool that was coerced into writing a web shell.

The takeaway for practitioners is to treat an LLM-backed application like a user or service account. Map what it can read and do, check whether its permissions exceed those of the people who use it, and identify any writable data source that feeds its context. LLM features should be on the list of targets during engagements, since they can open paths to credential access, collection, lateral movement and privilege escalation.