Episode 58: Cheng-Lin Yang and Lily Chen, CyCraft, FIRSTCON26 Speakers

### Overview In this episode of the First Impressions podcast, hosts Christian Riley and Martin McKay speak with CyCraft researchers Liddy Chen (Data Scientist) and George Yang (Data Scientist Director) about their upcoming FIRSTCON26 presentation, "One Poisoned Artifact Can Steer Your AI: How Robust Are Your LLM Assisted Security Workflows?" The talk explores the vulnerabilities that arise when security analysts rely on Large Language Models (LLMs) to browse the web and summarize incident reports, showing how an attacker can manipulate LLM-assisted workflows by placing a single crafted document online.

### Key Topics - **Evolution of AI Manipulation:** While earlier models could be tricked by simple social engineering or emotional prompts, modern LLMs feature higher baseline safety. Attackers must now focus on attention mechanisms to bias output generation. - **Indirect Prompt Injection:** Rather than sending prompts directly to the model, an attacker creates an external, poisoned document designed to catch the LLM's attention during external searches and alter its conclusions. - **Attack-Defense Asymmetry:** Modern AI security mirrors traditional cybersecurity "cat-and-mouse" dynamics. Offensive techniques continue to emerge and grow stronger, while generalizable defensive strategies have yet to converge.

### Technical Details The researchers describe a two-stage attack optimization strategy split across the primary components of retrieval-augmented LLM architectures: 1. **Attracting the Retriever:** One portion of the malicious document is optimized to attract the attention of the model's search/retrieval component, ensuring the document is selected from the web or an input corpus. 2. **Targeting the Generator:** A second portion is designed to influence the text generation component, forcing the LLM to alter its final summary, recommend malicious phishing URLs, or output incorrect claims.

During optimization, these documents often accumulate unnatural artifacts, appearing as gibberish or random-looking strings. Testing also revealed that attacks optimized for a specific model exhibit transferability across different LLMs, with smaller models generally proving easier to compromise. The technical foundations of this work were recently accepted at the ICML AI conference.

### Takeaways - **Inspect Noisy Artifacts:** Analysts and pipelines should not ignore gibberish strings or abnormal artifacts in raw logs and web content, as they may be optimized payloads designed to trigger model attention. - **Implement Contradiction Filtering:** Organizations should run secondary filters to detect whether a retrieved document contradicts the consensus of other gathered sources or drifts from the original query context before feeding it into generator models.