Extending Mesos for Large Scale Clusters

### Overview This presentation discusses extending Apache Mesos for large-scale production clusters, reflecting experiences from running distributed services and diverse workloads (such as microservices, batch processing, and storage systems) under high-reliability requirements. The talk details the operational challenges of deploying frameworks such as Apache Aurora and Apache Cassandra on Mesos, and highlights key technical architectural enhancements around event streaming, service discovery, allocation, and cluster automation.

### Key Topics - **Production Workloads and Frameworks**: Managing diverse workloads ranging from microservices to stateful storage engines like Cassandra on top of Mesos, as well as scheduler frameworks such as Aurora. - **Event Streaming & Integration**: Moving away from polling endpoints toward an HTTP-based event stream API to synchronize cluster and task state across external systems in real time. - **Service Discovery & Custom Executors**: Integrating discovery mechanisms (such as ZooKeeper) and leveraging custom Mesos executors to manage application lifecycles, health checks, and state transitions. - **Resource Management, Allocation, and Isolation**: Addressing cluster utilization, quota mechanisms, resource offers, preemption, oversubscription, and resource isolation (CPU, disk I/O, network) across multi-tenant workloads. - **Cluster Maintenance & Automation**: Operating Mesos at scale across multiple data centers, maintaining high availability during machine and agent failures, and automating rolling updates.

### Technical Details - **Mesos Event Stream Architecture**: External subscribers establish persistent HTTP connections to the Mesos master. Upon connection, the subscriber receives an initial snapshot of cluster state followed by incremental streaming updates (such as task launch, running, finished, or failed states). This reduces load compared to periodic endpoint polling. - **Stateful Services on Mesos**: Running stateful frameworks like Cassandra requires specialized handling for persistent storage, dynamic reservations, and failure recovery (e.g., node replacement, task restart, and data backup/restore workflows across multiple data centers). - **Scheduler and Executor Customization**: By coupling framework schedulers with tailored executors, custom logic can handle task registration, health checks, and direct status updates, improving coordination between discovery systems like ZooKeeper and Mesos masters. - **Allocation and Oversubscription**: Tuning allocator logic enables finer-grained quota enforcement, hierarchical roles, and optimistic resource offers. To maximize cluster hardware efficiency, oversubscription utilizes revocable resources while enforcing isolation controls to safeguard latency-critical workloads from batch workloads.

### Takeaways Extending Mesos to handle massive production scale requires decoupling state synchronization via event streaming, building robust automation around cluster maintenance, and tightly managing resource allocation and isolation. Tailoring framework schedulers and executors allows heterogeneous workloads—from stateless microservices to stateful distributed databases—to coexist efficiently on shared bare-metal infrastructure.