Top 10 Unexpected AI Systems Hacks that we did not see coming!

The talk opens with a rough map of the AI ecosystem, covering producers, deployers, users and bad actors, and notes that existing catalogues such as MITRE ATLAS and NIST AI RMF already list many known risks. The speakers then group their chosen attacks by who they hit: organizations, individuals and families, or infrastructure and markets. Each case is scored for likelihood, impact and how hard it is to prevent.

Organizational cases include a cloak meant to hide a person from thermal cameras guarding access points, spoken prompts that slip past audio guardrails, look-alike model names on public hubs that lead users to download tampered models, and poisoned feedback that lets fraud models miss real transactions. Consumer cases cover cloned celebrity images and voices used in romance and family emergency scams, smart glasses that profile strangers, and cloned voices that defeat single-factor bank verification. Macro cases include an automated trading flash crash and model extraction, linked to the market reaction after DeepSeek's release.

Suggested mitigations include guardrails tested across languages and phrasings, audits of model sources and data, circuit breakers for trading systems, multi-factor authentication rather than biometrics alone, tighter API key controls, family verification passwords, and AI inventories paired with data governance. The speakers note that no guardrail covers every edge case, so risks need prioritizing.