Building a Security Audit Logging System on a Shoestring Budget

The talk starts from a common problem: a fast-growing company adds services faster than it adds security staff, and intrusion detection events get lost when there is no central place to store them. The speaker argues for one centralized logging location and compares buying a commercial platform with building one. Buying brings a single vendor contact and possible discounts, but it risks lock-in, scaling claims that do not hold up, slow support on cheaper tiers, and features behind paywalls. Building gives control over data, retention and cost, but it takes weeks of effort and leaves the team responsible for patching and network access rules.

The middle of the talk covers architecture. A simple design sends events from log forwarders into OpenSearch, with a dashboard in front and per-source permissions so that each forwarder can only write to its own indices. A larger design adds a buffer ahead of ingestion to absorb failures and bursty traffic, a second forwarder to smooth the load, and bulk writes of a few megabytes per chunk. Regional clusters sit close to their data, and single sign-on with group mapping controls who can search what. On cost, the speaker recommends hot and cold data tiers, S3-backed storage for data that is rarely queried, spot nodes paired with replicas, shard sizes of roughly 20 to 50 gigabytes, and SSD sizing that accounts for IOPS limits tied to volume size.

The closing section covers upkeep. Index lifecycle policies can move data to cheaper storage automatically, configuration should be managed through GitOps, and periodic access reviews keep permissions tight. The main takeaway is that building can beat buying when a team accepts the ongoing operational work that comes with it.