Pwning User Phishing Training Through Scientific Lure Crafting

## Overview

In this presentation, Christian Dameff (Associate Professor at UCSD and Co-Director of the Center for Healthcare Cybersecurity) and Ariana Mirian (Security Researcher at Censys) examine the efficacy of simulated phishing tests and security awareness training. Phishing training is nearly ubiquitous—often mandated by cybersecurity insurance and compliance frameworks under the assumption that organizations can build a resilient "legion of human firewalls."

While historical laboratory studies suggested training was effective, recent real-world organizational data has cast significant doubt. The speakers argue that cybersecurity often deploys practices without rigorous evidence, contrasting this with medical research where treatments undergo multi-phase testing and randomized controlled trials (RCTs). To address this gap, the researchers conducted an evidence-based RCT to evaluate whether embedded training modalities genuinely alter user behavior.

---

## Study Design & Methodology

The study was modeled as a large-scale randomized controlled trial within a healthcare system: - **Participant Population:** Over 19,000 employees across three hospitals, spanning diverse job roles (doctors, nurses, sanitation engineers, cafeteria workers, and administrative staff). - **Duration & Cadence:** Conducted over eight months with monthly simulated phishing campaigns using 10 distinct lures to prevent pattern recognition. - **Experimental Cohorts:** Participants were randomized into five distinct groups: 1. **Control Group:** Received a generic `404` error page upon failing a lure, with no training delivered. 2. **Generic Static:** Received a fixed informational page containing standard phishing avoidance tips (e.g., verifying senders, inspecting headers). 3. **Generic Interactive:** Received general advice requiring interactive navigation and clicking through UI elements. 4. **Contextual Static:** Delivered training directly referencing the specific lure the user failed (e.g., password resets). 5. **Contextual Interactive:** Provided dynamic, interactive training tailored specifically to the context of the failed lure. - **Metrics Collected:** Click/failure rates per lure, homebrewed training engagement analytics (not provided natively by commercial vendor platforms), and elapsed time since the user’s last annual cybersecurity training.

---

## Key Findings & Lure Dynamics

- **Wide Variance in Failure Rates:** Failure rates across the 10 lures ranged dramatically from **1.8% to 30%**. Highly recognizable IT lures, such as Outlook password reset notifications, yielded the lowest failure rates (~1.8%), whereas internal organizational updates—such as a vacation policy change (30%) or a dress code violation (~27%)—had the highest failure rates. - **Simplicity vs. Complexity:** High-performing lures did not require complex designs, GIFs, or flashing elements; plain text relying on emotional or administrative salience was sufficient to prompt clicks. - **Control of the Metric:** Because the lure chosen determines the failure rate ("whoever controls the lures controls the failure rate"), organizations can manipulate reported success. Sending familiar technical alerts artificially deflates failure metrics, while organizational or policy-related lures induce significantly higher failure rates. - **Cumulative Exposure Risk:** Over the eight-month testing window, the cumulative failure curve showed that **over 50% of users fell for at least one phishing simulation**, demonstrating that with sufficient variety and time, most users will eventually fail.

---

## Implications & Takeaways

- **Flawed Evaluation Metrics:** Evaluating an organization's security posture using a single lure or static phishing simulation fails to capture true user vulnerability, as failure rates are heavily dependent on lure content. - **Reevaluating Punitive Measures:** Given that the majority of users fail simulations over time when exposed to diverse lures, organizations must avoid reprimanding or penalizing employees who click simulated phishing links. - **Adopting Evidence-Based Practices:** The security industry should transition away from vendor-driven claims and low-tier anecdotal evidence toward rigorous, medical-style empirical validation to ensure organizational time and resources are allocated effectively.