101: How To Break IPS & SIEM

Nicol Dankova recounts an incident from several years ago, when a security operations analyst had to write and deploy intrusion prevention rules for a newly disclosed vulnerability because the vendor had not yet released a signature. The team was under pressure during a busy period after a ransomware wave, so the rules were checked for matching behaviour and then rolled out to every IPS on a Friday evening.

The next morning, the SIEM was overwhelmed by IPS alerts, and the prevention systems in the core data center were blasting out events. The IPS settings had apparently reverted to defaults. Recovery was slow because there was no snapshot to roll back to, so the analyst had to export the list of changed rules and reset hundreds of them one by one. No on-call coverage existed for core security functions at the time, which made the outage harder to handle.

The takeaways are practical. Test how a new rule affects alert volume, not only whether it matches. Keep on-call coverage for critical security tooling, back up IPS and management configurations, and confirm whether the management console can roll back changes. Be cautious about making changes right before a weekend. The talk closes with a critique of the vendor's recovery mechanism, which the speaker described as unreliable.