The Fault in Our Metrics: Rethinking How We Measure Detection & Response

The talk opens with a problem many detection and response teams know well: metrics that were inherited, copied from past jobs, or built around alert counts, true and false positive ratios, and broad incident timers. The speaker, who has spent about a decade on call, admits to several of his own mistakes. These include tracking things his team could not change, chasing costly measurements when a rough one would do, and presenting specialist detail to executives who needed business impact.

To fix this, he suggests asking which question each metric answers, tying it to an outcome such as streamlining, threat awareness, vigilance or readiness, and filtering out time the team cannot reduce so that a result of zero signals no action is needed. Examples include measuring analyst time spent on false positives rather than their count, limiting coverage work to a top five threats list instead of the full ATT&CK matrix, and arranging metrics in a pyramid that links operational data to business impact. He also introduces a maturity model with 14 capabilities, each scored across process, tools, documentation and testing.

The closing guidance covers checking each metric for risky incentives, retiring metrics that no longer matter, and weighing data collection cost against the time a team actually has. When replacing an entrenched metric, he recommends expecting pushback and using a maturity view to tell the new story. Questions covered building the maturity graph and weighting ATT&CK test coverage.