Hyper-scale Detection and Response

The speakers, from eBay's security analytics and detection engineering teams, start with the problem. Licensing fees for a commercial SIEM grew painful once daily ingestion reached about 20 terabytes. Their infrastructure is mostly on-premises with many custom log sources, and older syslog-style transport was losing logs, so both cost and reliability had to be addressed.

They describe the replacement, called Argus, which pairs Kafka for reliable transport, Apache Flink for stream processing and Hadoop for inexpensive large-scale storage. Logs reach the pipeline through an internal monitoring platform built on Elastic Beats agents. Detections are written as Flink SQL, optionally extended with Java user-defined functions, and saved as jobs that run continuously and push alerts to ServiceNow. A short demo showed a detection engineer building a rule and an incident responder opening a prepopulated drill-down query against Hadoop. Templates keep the analyst workflow simple.

The results cited were ingestion growing to about 55 terabytes per day, broader coverage, lower licensing spend and less dependence on a vendor. The team plans to retire the third-party SIEM fully and explore machine learning on the larger datasets. Their advice was to try unfamiliar open-source stacks, which proved easier than expected, and they noted that Druid, Elasticsearch and S3 were also evaluated before these three components were chosen.