Pentesting By The Numbers: What Data Analysis Tells About What We Do by Charl Van Der Walt

### Overview In this talk, Charl van der Walt (Head of Security Research at Orange Cyber Defense) explores whether vulnerability scanners and penetration testers are actually helping organizations defend against real-world threats. Using an extensive dataset spanning millions of vulnerability scanning findings and roughly 1,500 penetration testing reports, van der Walt investigates the disconnect between vulnerabilities actively exploited by threat actors and those discovered and reported by security professionals.

### Key Topics - **Vulnerability Management Overload:** With ~18,000 new CVEs annually and an average of 18–19 actionable findings per host, organizations struggle to patch more than 5% to 20% of known flaws, even though only an estimated 2% to 7% are ever exploited in the wild. - **Data Ingestion & Methodology:** Vulnerability scan data from scanners like Qualys and Rapid7 was aggregated via Nucleus across 41,000 hosts (~3 million findings). Penetration test reports (Word documents) were unzipped, parsed via XML, anonymized, and processed using spaCy NLP for Named Entity Recognition (NER) to extract CVEs and metadata across ~1,500 assessments. - **Remediation Timelines & Demographics:** Across scanning data, remediating a vulnerability takes an average of 215 days (72% take >30 days, 52% take >90 days). Critical vulnerabilities are resolved ~35% faster, yet still average over six months. Furthermore, 47% of reported CVEs across scanning and pen testing were over five years old. - **Comparative Exploit Datasets:** The speaker compared customer vulnerability exposure and pen tester findings against three threat intelligence benchmarks: 1. *Ivanti Ransomware Top 10:* Only 5 of the top 10 appeared in customer scans, and only 1 (Log4j) was ever exploited by pen testers across 1,500 assessments (exploited exactly once). 2. *CISA Known Exploited Vulnerabilities (KEV):* 52% of the ~860 KEV vulnerabilities never appeared across 41,000 customer hosts. Of the remaining KEV vulnerabilities present, pen testers reported only 7 unique CVEs. 3. *FIRST Exploit Prediction Scoring System (EPSS v3):* EPSS demonstrates that targeting a low threshold (0.088) achieves 82% exploit coverage by patching only 7.3% of vulnerabilities (45% efficiency), compared to CVSS >= 7 requiring 58% of all CVEs to be patched (3.9% efficiency). Evaluating vulnerabilities with an EPSS score > 0.03 present in customer environments, pen testers reported zero of them. - **Trends Over Time:** Between 2018 and 2022, penetration tests exhibited a downward trend in findings per day (measured in CVSS per day) and a drop in serious findings from 71% in Q4 2018 down to 33% in Q1 2022. Potential causes discussed include improved base security from modern cloud frameworks, recurring customer maturation, and remote testing constraints over VPN post-2020. - **Complex Attack Realities:** Referencing the LastPass breach, van der Walt highlighted how real-world intrusions often link disjointed components (credential compromise, shared cloud backups, developer home networks, and targeted software flaws like Plex CVE-2020-5741) that isolated scanning and testing rarely capture or prioritize.

### Technical Details - **Metrics:** Introduced "CVSS per day" (total CVSS points divided by man-days) to normalize pen testing output. Internal tests yielded the highest volume (~8 CVSS points/day), while red teaming yielded the lowest (~2 CVSS points/day) due to fewer, higher-severity findings. - **Scoring & Distributions:** Vulnerability scanners and pen testers both heavily report medium-severity issues (peaking around CVSS 4–5). Pen testers moderate ratings with context, rarely assigning maximum severity scores and occasionally assigning 0, whereas scanners assign higher critical volumes.

### Takeaways Security assessments often suffer from an emulation disconnect, echoing Haroon Meer's critique that pen testers often emulate other pen testers rather than real threat actors. While organizations invest heavily in web application gymnastics and chasing CVSS scores, real attackers repeatedly rely on credential stuffing, phishing, password reuse, and specific unpatched perimeter services. Security programs must re-evaluate prioritize-by-CVSS models and leverage data-driven intelligence like EPSS to focus on the small fraction of vulnerabilities that attackers actually exploit.