Processing Billions of Events in Real-Time with Heron by Karthik Ramasamy, Twitter

### Overview In this presentation, Karthik Ramasamy introduces Twitter Heron, an open-source real-time streaming engine developed and run in production at Twitter for over two years. Faced with severe scalability, operational, and debugging limitations while running Apache Storm across thousands of nodes—including 19 specific problems documented in a SIGMOD paper—Twitter built Heron as an API-compatible replacement. Heron was designed to handle Twitter's vast real-time event processing demands, such as calculating real-time trends, conversation bubbling during major sporting events, ad relevance injection, search indexing within hundreds of milliseconds, fraud/abuse detection, and real-time operations and BI.

### Key Topics - **Architecture and Topology Execution**: Similar to Storm, Heron organizes computation as a directed acyclic graph (DAG) of spouts (data sources) and bolts (computational logic). However, Heron separates execution into dedicated Linux cgroup containers managed by existing schedulers (such as Mesos/Aurora, Marathon, YARN, and Slurm). Container 0 runs the Topology Master, which advertises discovery endpoints via ZooKeeper, creates the physical execution plan, and coordinates container allocation. Data containers run Stream Managers and individual process-isolated Heron Instances (`I1`, `I2`, etc.), alongside Metrics Managers. - **Process Isolation & Language Support**: Moving away from Storm's multithreaded Clojure model, Heron uses isolated single-task OS processes written in mainstream languages (C++, Java, Python). This isolation prevents garbage collection cross-contamination, simplifies heap dumping and profiling, and allows multi-tenant execution alongside critical services. - **Componentized "Micro Streaming" Engine**: Inspired by the Mach microkernel architecture, Heron keeps only core primitives (IPC, scheduler abstraction, distributed state) in its core engine, delegating scheduling and state management to external pluggable systems (e.g., ZooKeeper, HDFS, S3, Aurora, YARN) without modifying core engine logic. - **Handling Stragglers via Backpressure**: Multi-tenant distributed clusters face stragglers from bad hosts, execution key skew, or underprovisioned resources. Heron avoids Storm's unpredictable data-dropping behavior by employing a "slow down sender" strategy. Stream Managers detect processing bottlenecks using high and low watermark buffers, broadcasting initiate and relief backpressure control messages that deactivate socket reads at the spouts via `libevent` event loops.

### Technical Details - **Resource Consumption and Breakdown**: Profiling production topologies revealed that deserialization and data ingestion dominate processing costs. In a topology filtering and exploding 6,200 million tuples per minute down to Redis cache writes, spouts consumed 85% of assigned CPU cores, while bolts consumed 9% and Heron engine overhead took only 7%. Further breakdown showed deserialization accounted for 63% of spout time, and Kafka fetching accounted for roughly 25%. - **Efficiency Gains**: Heron demonstrated up to a 10x reduction in CPU resource usage compared to Storm for production workloads (e.g., reducing core usage from 370 cores to 30–40 cores for a real-time active user engagement topology) while delivering higher throughput and lower latencies. - **Failure Recovery**: If a container dies, schedulers automatically replace it; Stream Managers reconnect using unique identifiers and ZooKeeper metadata, prompting the Topology Master to reconstitute and redistribute the physical plan automatically.

### Takeaways Heron demonstrates that decoupling the streaming engine into isolated processes and modular plug-and-play components significantly enhances predictability, debuggability, and resource efficiency at scale. Backpressure with buffered watermarks provides predictable processing and fast spike recovery without sacrificing data accuracy. Active ongoing open-source work on Heron includes dynamic auto-scaling via CLI (`heron update`), machine learning-based root cause diagnosis for faulty nodes, SQL support, and direct high-performance transport alternatives.