GeeCON 2019: Kamil Owczarek - Data Streaming: Which horse to bet on?

### Overview In this talk from GeeCON 2019, Kamil Owczarek examines the landscape of stream processing engines, addressing why organizations migrate toward data streaming and how to choose the right framework for specific use cases. Drawing from consulting experience as head of a Big Data practice, the speaker explores core streaming mechanics—including time handling, windowing types, joins, aggregations, and failure recovery—and compares four prominent technologies: Akka Streams, Kafka Streams, Apache Flink, and Apache Spark Streaming (Structured Streaming), with brief commentary on Apache Beam.

### Key Topics - **Drivers for Streaming**: Real-time connected cars, financial and trade analytics, IoT telemetry, and user interaction/clickstream analysis (including fraud prevention and targeted advertising). - **Processing Paradigms**: Pure event-by-event streaming versus micro-batching (used by Spark), where intervals are kept minimal but latency remains higher than true streaming. - **Notions of Time & Windowing**: Processing time, event time (embedded in records to handle out-of-order data), and ingestion time. Windowing models covered include tumbling (fixed, non-overlapping), sliding (overlapping, useful for anomaly detection and trends), and session windows (gap-based activity tracking). - **Stream Operations**: Stream-to-stream joins (which require explicit time boundaries to prevent unbounded memory growth) and windowed aggregations. - **Evaluation Metrics & Reliability Features**: Latency versus throughput trade-offs, lateness handling (watermarking), checkpointing (local filesystem vs. distributed storage/external persistence), savepoints (Flink), and third-party data integration capabilities.

### Technical Details - **Akka Streams**: Operates on the Akka actor model (Scala/Java), offering massive concurrency, reactive backpressure, granular low-level I/O control, and extensive connectivity via the Alpakka project. It lacks native out-of-the-box watermarking and native checkpointing, and lacks explicit built-in session windows, but provides very low per-record latency. - **Kafka Streams**: Tailored specifically for reading and writing to Apache Kafka. It runs lightweight embedded instances on the JVM, supports watermarking, local checkpointing, and session windows, but lacks native connectors to outside datastores without integrating auxiliary tooling like Kafka Connect. - **Apache Flink**: True event-driven cluster processing platform with support for watermarking, local and distributed checkpoints, user-triggered savepoints (enabling seamless cluster upgrades and maintenance), and native session windows. It balances low latency (approaching Kafka Streams) with scalable throughput. - **Apache Spark Streaming (Structured Streaming)**: Micro-batch architecture that sacrifices ultra-low latency (typically around one second minimum, up to multiple tens of seconds for complex computations) in exchange for massive throughput, cluster-wide caching, and unified integration with Spark ML and graph modules. It supports distributed checkpoints and watermarking, but lacks explicit session window constructs. - **Apache Beam**: Serves as an abstraction layer over engines like Spark and Flink or Google Cloud Dataflow, but does not target lightweight JVM frameworks like Akka Streams or Kafka Streams, and suffers from inconsistent cross-engine feature parity.

### Takeaways Owczarek emphasizes that there is no universal "silver bullet" in streaming architectures: 1. **Ultra-Low Latency / IoT**: Akka Streams (especially via protocols like MQTT or raw TCP) or Kafka Streams are preferred when immediate reaction times are critical and records arrive individually. 2. **Extensive Data Integration**: Apache Flink or Apache Spark are optimal when out-of-the-box connectors to multiple diverse databases, message queues, and distributed file systems are required. 3. **User Behavior & Clickstreams**: Kafka Streams is ideal due to native session window support, fast processing, and built-in state management. 4. **Complex Iterative Analytics & Machine Learning**: Apache Spark Streaming excels when massive aggregation, distributed caching, and built-in ML libraries outweigh sub-second latency requirements.