Martin Holste

### Overview In this presentation, Martin Holste examines the limitations of complex security analytics and outlines a modular, practical approach to threat detection. While advanced machine learning and analytics are often promoted as game-changing solutions, Holste argues they often fail to deliver standalone value in security operations. Drawing parallels to retail recommendation systems, he explains that security operates in a high-risk environment where false positives and false negatives carry steep costs. Rather than attempting to deploy monolithic, "black box" machine learning models that generate overwhelming alert volume, security teams should focus on building simple, explainable, and combinable analytical building blocks.

### Key Topics - **Challenges of Applying Data Science to Security:** Unlike low-risk retail analytics (e.g., product recommendations), cybersecurity operations face high costs when analytics fail. Data scientists and security analysts frequently lack a shared framework, leading to disconnects during feature engineering and handoffs. - **Impracticalities of Monolithic Analytics:** Specific analytics (such as hunting solely for DNS tunneling) can have high opportunity costs compared to widespread exploit kits. Additionally, extensive whitelisting (e.g., in data exfiltration), cost-prohibitive computational scaling (such as N-by-N clustering across billions of events), and black box issues (like unexplainable neural networks) severely impede real-world sock operations. - **Data Quality and Normalization:** Holste references a 2014 Google Stanford lecture noting foundational difficulties in log normalization, taxonomy discrepancies (e.g., username formats), time stamps, multi-line logs, and lack of clean ground-truth training data in incident response. - **Signal Fatigue and Overfitting:** Even an analytic with 99% accuracy can flag 1% of benign enterprise activity—amounting to hundreds of alerts daily for normal, idiosyncratic human behavior. Continuously tweaking an analytic to fit edge cases often results in an overspecialized rule rather than a robust analytic model.

### Technical Details - **Hallmarks of Practical Analytics:** To be viable, an analytic should be easy to compute, easy to understand, and easy to convey. It should present objective facts rather than unexplainable opinions, avoid requiring pristine ground-truth data, and output standard events that feed back into existing workflows. - **The UNIX Philosophy of Analytics (Set Intersections):** Holste proposes constructing modular analytics that perform one task well, treating them as pipeline stages. By intersecting multiple sets (such as "strange pairs"), noisy or uninteresting lists—such as users downloading executables, users on vacation, or top browsing volumes—become high-fidelity indicators when correlated over specific timeframes. - **Credential Misuse Demonstration:** To demonstrate set intersection, Holste illustrates analyzing credential misuse through faceted queries and visualization. Correlating categories such as service accounts, power users, and first-time logins against source code repositories narrows thousands of candidate events down to a single actionable investigation. - **Effective Visualization and Dashboards:** Holste critiques traditional cluttered dashboards and pie charts that require constant hovering. He advocates for clear visualizations that minimize wasted screen space, citing Edward Tufte's *Beautiful Evidence*, Sankey diagrams for multidimensional relationship tracking, and time-series plotting inspired by Galileo's tracking of Jupiter’s moons.

### Takeaways Holste concludes that the most effective security analytics resemble basic reporting and statistical summaries rather than opaque algorithms. Practical analytics should rarely be expected to identify malicious activity in isolation; their true value lies in serving as factual building blocks that can be piped together into standard analyst workflows. By adopting simpler analytical primitives, security teams can bridge the gap between data scientists and SOC analysts while mitigating signal fatigue.