The 8th Annual Black Hat USA NOC Report

Overview: Neil Wyler ("Grifter"), Global Lead of Active Threat Assessments at IBM X-Force, and Bart Stump, Senior Systems Engineer at NetWitness, deliver the 8th Annual Black Hat USA Network Operations Center (NOC) Report. Celebrating Wyler's 20th year running the conference network, the presenters provide an architectural and operational walkthrough of building and monitoring one of the world's most hostile and high-density temporary production enterprise networks. The talk details the hardware infrastructure, partner ecosystem, live packet analysis workflows, and operational challenges surmounted during the hybrid 2022 event.

Key Topics: - High-density conference network architecture and upstream connectivity via Lumen - Full-fidelity packet analysis and log aggregation using NetWitness, Gigamon, and Palo Alto Networks - Emergency hardware replacement and rapid wireless migration using Cisco Meraki - Operational telemetry dashboards, threat hunting, and the open-source "pew-pew" traffic visualization map

Technical Details: The presentation details the physical and logical architecture supporting the Black Hat USA network, which is constructed and operated by a volunteer core of 15 Technical Associates alongside partner engineering teams. Dual redundant 10-gigabit upstream circuits provided by Lumen (formerly CenturyLink) connect the venue directly to points of presence in Denver and Sunnyvale. These upstream pipes terminate into redundant core switching hardware, which connects directly to high-availability Palo Alto Networks next-generation firewalls. The firewalls serve as the Layer 3 gateways for every VLAN and subnet across the venue, enforcing routing and segmentation across conference areas, classrooms, vendor floors, and broadcast infrastructure.

To ensure comprehensive visibility across the untrusted network, Gigamon network TAPs duplicate and distribute line-rate packet streams directly to specialized security tooling without impacting network performance. NetWitness deploys enterprise packet capture and log analytics appliances on-site to capture, store, and index every raw packet traversing the wire, providing full-fidelity packet analysis, endpoint telemetry, and automated orchestration. IronNet supplies behavioral network traffic analysis, while Cisco contributes Threat Grid for dynamic malware sandboxing and Cisco Umbrella for DNS-layer visibility. Due to severe post-pandemic supply chain shortages, a primary hardware switching partner withdrew just 26 days before the Singapore conference; Cisco rapidly stepped in to deploy Meraki switches and wireless access points, which were battle-tested overseas before full-scale deployment at the Black Hat USA show.

Inside the NOC, analysts rely on live operational dashboards to monitor anomalies in real time. The centerpiece display features an open-source visual traffic map ("pew-pew map") that tracks the 50 most chatty hosts on the network. The map renders dynamic protocol vectors—using green for TCP, red for UDP, and white for ICMP—where stream thickness ("lightsaber" style) indicates heavy bandwidth consumption and sudden bursts ("fireworks" patterns) signify sweeping network scans. When anomalous volumetric flows or suspicious scanning behaviors appear on the visualization, NOC operators immediately dispatch investigation tickets to the NetWitness engineering team to perform deep packet payload analysis and determine the nature of the communicating endpoints.

Key Takeaways: Operating high-risk network environments requires complete Layer 1 through Layer 7 visibility, utilizing hardware network TAPs to feed full-packet capture systems without introducing latency or points of failure. Network operations teams should deploy real-time visual telemetry and chatty-host trackers to rapidly detect reconnaissance scanning and anomalous data transfers before digging into deep packet analysis. Robust incident response and event continuity depend on establishing vendor redundancy and agile deployment partnerships to overcome sudden supply chain disruptions.

(AI summary — Gemini Flash)