GeeCON 2024: Viktor Gamov - How do you query a stream?

### Overview In this talk from GeeCON 2024, Viktor Gamov explores the architectural challenges and architectural trade-offs involved in querying real-time streaming data, particularly when using Apache Kafka as the underlying streaming substrate. He traces the evolution from monolithic databases using SQL for CRUD operations to modern distributed streaming systems that capture event history in append-only logs. Throughout the presentation, Gamov surveys several architectural patterns and tool categories for querying streams, comparing transactional approaches against real-time analytical workloads.

### Key Topics - **Evolution of Data Architecture**: The shift from monolithic databases handling row-level OLTP operations to event-driven architectures, CDC (change data capture), and append-only distributed logs in Kafka. - **Out-of-the-Box Kafka Tooling**: How Kafka Connect and Kafka Streams provide baseline options for data movement, stream transformations, and stateful processing via KTables. - **Streaming SQL and Streaming Databases**: Emerging specialized technologies such as Materialize, Delta Stream, RisingWave, and ksqlDB that treat continuous streams as queryable SQL tables or materialized views. - **Cloud Data Warehouses & Data Lakes**: Architectural characteristics and trade-offs of systems like BigQuery, Redshift, Snowflake, Apache Iceberg, and S3-based data lakes for analytical querying. - **Real-Time OLAP Databases**: Systems like Apache Pinot, ClickHouse, and Apache Druid designed to serve sub-second analytical queries at massive concurrency on real-time data.

### Technical Details - **Kafka Connect**: Acts as a framework for building data pipelines without custom code, reading from sources and sinking to transactional databases like PostgreSQL. While simple for small-scale analytics, it introduces infrastructure complexity and is constrained by how much analytical load an OLTP database can sustain. - **Kafka Streams & KTables**: A Java framework that runs embedded inside applications (e.g., Spring Boot or Micronaut) without dedicated cluster management. It models stream-table duality using in-memory KTables representing state. Querying is possible via interactive query APIs (such as gRPC or REST) over local RocksDB/in-memory key-value state, but updating query logic requires code changes and application redeployments. - **Streaming SQL Engines**: Tools such as Materialize implement persistent, incrementally updated materialized views accessible via the PostgreSQL wire protocol. Delta Stream provides serverless stream processing with a "bring your own cloud" model. RisingWave implements distributed SQL using C++ with PostgreSQL wire compatibility, and ksqlDB offers push/pull queries over Kafka streams. - **Real-Time OLAP**: Platforms like Apache Pinot (utilizing Apache Calcite for SQL parsing) ingest directly from Kafka partitions to provide low-latency queries across high query concurrency. They leverage specialized indexing—including range, sorted, inverted, and JSON path indexes—to support high-throughput, user-facing analytical workloads without reparsing nested data.

### Takeaways There is no single "definitive" tool for querying streaming data; architects must evaluate trade-offs based on use case constraints. Key selection criteria include query latency requirements, data freshness, infrastructure operational overhead, analytical versus transactional query patterns, and active community support. When data volumes grow into hundreds of gigabytes or require user-facing low-latency queries, specialized real-time OLAP databases or streaming SQL engines become preferable to routing streaming events into traditional relational databases.