Fragmentation of CTI: The Deck is Stacked Against the Defenders

The talk introduces STPA-Sec, a systems-theoretic hazard analysis method extended to cover security, and argues that traditional failure analysis misses how AI systems can go wrong without any component breaking. The presenters frame safety as a control problem and walk through the core vocabulary: losses, system-level hazards, and the four categories of unsafe control action.

Two case studies follow. The first models an AI coding assistant, with a developer, an agent plugin, an IDE, logging, and a training pipeline acting as controllers. The hazards they list include vulnerable code accepted without review, proprietary code leaking through feedback logs, and updates that introduce regressions. The second covers an analyst using an LLM to digest open-source intelligence, where confident fabrication becomes the central hazard. A later point adds a repository README carrying adversarial instructions that an AI tool could follow after cloning, which widens the modeled attack surface.

The method then moves from hazards to mitigation: deriving formal safety and security constraints, building behavior models, generating test cases, and wargaming adversary actions with red and blue teams. The closing takeaways stress bringing risk, compliance and governance into design early and iterating across the lifecycle. In the Q&A, the presenters share examples of a radar gain control that suppressed real targets and adversarial patches that could disable a camera-based detection system, arguing that robustness testing must feed back into design.