Why Your Security Metrics Suck, And How to Fix This

The talk opens with two confusing charts shared online to show how easily metrics mislead when they lack context. The speaker argues that many security metrics are vague, use loose words like "better" or "easier," skip baselines, and are presented without a clear story. He recalls early work collecting monthly device counts that never explained anything, and points to denominators, such as network segment totals, that people often guess when nobody knows the real number.

The middle of the talk covers what good metrics are for. The speaker describes three purposes: setting a narrative, showing progress against a fixed baseline, and demonstrating cause and effect without falling into the correlation trap. He stresses concrete language, consistent formulas, showing the math, and avoiding feelings or cherry-picked trend lines. He also explains that detail should change by audience, from technical staff to leadership to the board, and contrasts prevention with detection and response. Error rates, staff cuts and budget freezes are offered as examples of measures that can show real business impact.

The close gives a checklist of questions to ask before presenting a metric: what the message is, which variables sit on each axis, whether a non-expert would understand it, whether it is quantitative, and whether the framing gives needed context. The takeaway is that a metric should let an outsider grasp its point without having to be told how to read it.