"Chasing the Stream Processing Utopia" by Kartik Paramasivam

### Overview Kartik Paramasivam, manager of near real-time data processing and streams infrastructure at LinkedIn, outlines the vision for an ideal "utopia" in stream processing. At LinkedIn, streaming infrastructure processes more than a petabyte of data and trillions of messages daily using Apache Kafka for pub/sub messaging and Apache Samza for stream processing. Stream processing is employed across varied domains including social computing, bot detection, application monitoring, and security monitoring. Paramasivam identifies the fundamental requirements for reaching an ideal stream processing world: empowering developers to build applications of any complexity, operate at any scale, ingest data from any source, code in their language of choice, deploy in any environment, and leverage robust tooling.

### Key Topics - **Application Complexity and Patterns**: Examining the shift from simple stateless operations (filtering, transformation) to complex stream-table joins and stateful aggregations (windowing, tumbling windows). Addressing accuracy challenges such as exactly-once processing (EOP) across ingestion, processing, and storage sink tiers. - **Scaling Ingestion and Clusters**: Operating large-scale distributed message queues using Apache Kafka, managing bare-metal cluster sizing (limiting clusters to 125–150 nodes), multi-cluster federation, disk failure resilience without broker crashes, and automated cluster rebalancing with Kafka Cruise Control. - **Scaling Stream Processing and State Management**: Mitigating I/O bottlenecks in data access through asynchronous calls, batching, caching, and local materialized caches (via change data capture systems like Brooklin). Implementing high-performance local state storage using RocksDB, incremental checkpointing, and host affinity in Apache Hadoop YARN. - **Unifying Batch and Stream Sources**: Using HDFS as a stream input source with an "end of stream" concept to seamlessly execute streaming applications as batch jobs for model evaluation and historical experimentation. - **Accessibility through Polyglot APIs and SQL**: Lowering the barrier to entry via Apache Calcite SQL, converting legacy Hive and Pig scripts to stream processing topologies, and integrating Apache Beam for Python support. - **Ecosystem Tooling and Observability**: Bridging the tooling gap between batch and stream processing with pipeline visibility, schema management, lag monitoring, and health tracking using tools like Burrow and Kafka Monitor.

### Technical Details - **State and Data Enrichment**: When joining streaming data with databases like LinkedIn's homegrown Espresso, relying on remote RPC lookups risks denial-of-service on upstream databases. Samza introduces a remote `Table` abstraction incorporating async I/O, caching, and rate limiting, as well as a local table approach backed by Brooklin change capture streams and partitioned local RocksDB instances. Local state yields roughly 100x to 200x faster lookups compared to remote databases. - **Reliability and Host Affinity**: Samza coordinates with YARN to guarantee container host affinity, ensuring that stateful tasks reattach to their existing RocksDB data upon restart or upgrade, while replaying state changes from Kafka changelog topics only when needed. Full-snapshot checkpoints fail at gigabyte and terabyte scale, making incremental checkpoints mandatory. - **Exactly-Once Semantics**: Handled across three stages: (1) ingestion deduplication using deterministic client IDs, (2) pipeline processing deduplication using techniques like Chandy-Lamport distributed snapshotting rather than heavier transaction models, and (3) storage persistence using idempotent writes or transactional database capabilities. - **Ingestion Operational Engineering**: LinkedIn avoided RAID redundancy overhead on Kafka by updating Kafka brokers to tolerate individual disk failures. Cluster operational overhead is automated via Kafka Cruise Control, which balances traffic across brokers without saturating network interface cards (NICs).

### Takeaways Achieving stream processing utopia requires moving beyond basic stream-processing primitives toward higher-level abstractions. Critical enablers include declarative SQL (via Apache Calcite), unified batch and streaming models (via Apache Beam and HDFS end-of-stream triggers), managed stream processing as a service with automated resource tuning, and self-healing cluster and pipeline operational tools such as Kafka Cruise Control, Burrow, and Kafka Monitor.