Cassandra Serving Netflix @ Scale

### Overview At Netflix, Apache Cassandra serves as a critical persistence store for nearly all major features, including member viewing history, ratings, metadata, billing, and recommendations. The deployment spans hundreds of clusters, tens of thousands of nodes, and petabytes of data running fully on AWS (predominantly on i2 instance types such as i2.2xlarge, i2.4xlarge, and i2.8xlarge). Running a stateful distributed database at this scale in a cloud environment introduces operational hurdles across monitoring, maintenance, resiliency, and open-source benchmarking. This talk details the architectural tooling Netflix engineered to solve these challenges.

### Key Topics - **Transitioning from Pull-based to Push-based Streaming Monitoring**: Traditional cron-based health checking (e.g., using Jenkins to poll instances) failed at scale due to lack of cluster state awareness, transient cloud network hiccups, and failure under JVM/connection pressure. Netflix replaced this with a streaming architecture built on Mantis (an operational streaming platform on Apache Mesos). Instances push heartbeats every 10–20 seconds to isolated regional "local thing aggregators," which evaluate gossip state, client protocols (Thrift/binary), JMX JVM metrics, and hardware health into a compact regional score. A centralized "global thing aggregator" runs a finite state machine that accounts for cloud instance lifecycles (such as 24–48 hour token range data transfer during node replacement) to prevent false alerts, reducing on-call alert pages from 300 per day to under 100 per week. - **Automated Lifecycle Management with Priam**: To eliminate manual maintenance and shell scripting across thousands of nodes, Netflix built and open-sourced Priam, a sidecar process co-located on every Cassandra instance. Priam automates cluster and node bootstrapping, automated non-vnode token assignment to prevent hotspots, configuration management, automated backup/recovery, and metric/heartbeat reporting via REST APIs and JMX. - **High Resiliency and Multi-Zone Architecture**: Deployments avoid Cassandra virtual nodes (vnodes) due to historical multi-zone replication awareness issues. Instead, Priam distributes tokens systematically across three AWS Availability Zones (AZs) per region with a replication factor of 3. Operating clusters at two-thirds capacity guarantees that even an entire AZ loss will not degrade 99th-percentile read/write latencies. Regular Chaos Kong exercises test instance and zone terminations, while cross-region traffic shifting handles regional failures. - **Production-Realistic Benchmarking with NDниBench**: To certify new Cassandra versions, Java drivers, or cloud AMIs prior to deployment, Netflix developed and open-sourced NDниBench (Netflix Data Benchmarking). NDниBench executes dynamic configurations and realistic traffic patterns directly inside the production-like cloud ecosystem without introducing measurement overhead.

### Technical Details - **Ecosystem Integration**: Alongside Priam and Mantis, Cassandra instances interface with Winston (automated remediation), Atlas (metrics/alerting), Jenkins, log analysis via ELK, and Cassandra Reaper. - **Capacity and Autoscaling**: About 20% of scaling operations are automated using prediction bots and outlier detection, with further automation in progress.

### Takeaways Operating large-scale stateful systems in the cloud requires decoupling state tracking from point-in-time checks, shifting from polling to continuous stream processing with state machine awareness, isolating management into dedicated sidecars, and strictly testing through automated chaos and dynamic production-mimicking benchmarks.