A Serverless SIEM: Detecting All Baddies

The talk explains why a detection and response team chose to build its own SIEM instead of running an existing one. The speakers described the pain points of their earlier setup: patching a self-hosted cluster, slow or failing searches, weak scaling during large attacks, and the need for logging agents on every server. Their requirements covered global log intake, efficient and safe transfer, fast search, easy rule authoring, industry-standard rule formats, and low cost.

The architecture runs on Google Cloud with no servers to manage. Logs arrive through several paths: writes to storage buckets that trigger processing, native exports into BigQuery, and scheduled pulls from SaaS audit APIs every few minutes. Detections come in two styles: stream rules written in CEL that evaluate each event, and batch rules written as SQL on BigQuery schedules. Rules live in version control with Sigma-style formats and editor tooling. A notification component deduplicates repeated alerts into a single ticket and routes them to chat, Jira, or PagerDuty. An Airflow-based automation layer enriches alerts and can ask users to confirm suspicious activity.

The speakers put storage cost at roughly 50 to 70 cents per gigabyte with a year of searchable retention, and noted that about half of their spend came from an optional hot-search logging tier. Development, staging, and production environments are all managed with Terraform. Next steps include log normalization, threat intelligence integration, and more understandable anomaly detection. In the Q&A they warned about cloud billing traps, such as unpartitioned BigQuery tables, and said open-sourcing parts of the project is on the roadmap.