Social Engineering Your Metrics: Using Data Science to Provide Value in Reporting

The talk starts by reviewing common security metrics, including password length and reuse, breach dwell time and return on investment. The speaker argues that social engineering reports lean too heavily on email opens, clicks and captured credentials, which say little about how an organization actually responds. Stolen credentials, for example, usually point to missing controls such as password policy or multi-factor authentication rather than to individual user failure.

The core proposal is to measure the time between events rather than raw counts. Ratios such as open-to-click show how many recipients move from one step to the next, while distance metrics such as open-to-report or click-to-report show how quickly suspicious mail reaches the security team and where improvement can be measured across engagements. A containment example illustrated the point: when a user who clicked reported immediately, the response window shrank from many minutes to a short interval. The speaker also covered comparing results per client, across sliced engagements and across the full population, along with sample size, mean, median and standard deviation. Regression and cluster analysis were presented as ways to compare outcomes, with the caveat that logs and synchronized timestamps are needed to trace each step.

The closing section made the case that advanced methods such as recurrent neural networks are not required to start. Basic statistics, a consistent program run over time and a clear baseline are enough for most teams. Practitioners were advised to present results to management as a concept of operations, tailor the format to the audience, and remain aware that statistics can be manipulated, so data should be collected and analyzed ethically.