AI Under the Hood: Unmasking Hidden Threats

The talk starts from the idea that AI systems need security testing suited to their risk, size and application, much as a heavy truck's brakes differ from a small car's. It covers adversarial inputs, such as altered stop signs that can fool self-driving perception models, and data poisoning, where manipulated training images cause a classifier to mislabel objects. It also explains why AI security is hard to test: there are no predefined test cases, and models are usually tested as a whole rather than in parts.

The middle section describes a process for securing AI applications: inventory the components, rank them by criticality, threat model them, and then verify them. The main focus is a threat modeling library called Guardrail. The team reviewed 14 published AI threat resources and extracted more than 400 threats, then removed duplicates and items that were not AI-specific. The remaining threats were turned into questions, grouped into a baseline set plus add-on sets for continuous learning and applications that use user data, and refined through expert interviews.

The talk closes with the scope split between manual threat modeling, which is the focus, and automated adversarial robustness testing, which was left out for now. Future work includes systematising adversarial attacks, reviewing testing tools, and looking for CVEs tied to adversarial attacks. For practitioners, the takeaways are to first ask whether AI is needed at all, to weigh build versus buy with their risk appetite in mind, and to use a shared question set when analysing AI-specific threats.