Security Monitoring And Response In Large Linux Environments

The speakers describe a security monitoring project for a large enterprise running more than 10,000 systems, mostly Linux, with over 100,000 clients connecting to them. Asset inventories were out of date, and cloud and container hosts appeared and disappeared constantly, so a central asset list could not be trusted. Their raw audit output was too noisy to be useful in Splunk, so they wrote a post-processor that turned it into structured JSON.

The Laurel tool runs on each endpoint and adds process ancestry and custom labels based on its configuration. A labelled web application process that spawns a shell becomes an easy signal to hunt for. The team also evaluated commercial EDR products and found Windows-first designs with weak Unix handling, opaque detection logic and limited custom rules. They chose Velociraptor for Linux server response, running clients on every server with a distributed master and minion nodes behind a TCP load balancer. The shared NFS state became a bottleneck they were still working on. Velociraptor artifacts gather package, port and service data to label hosts, run integrity checks and YARA scans, and work with the SIEM, whose alerts trigger automated investigations.

The main lessons were that enterprises should contribute back to open-source projects, since maintainers lack their operational experience, and that the setup scaled to roughly 10,000 to 20,000 systems with more hardware. The team plans to move its sensors to eBPF and publish more of its artifacts.