From Zero to Production

### Overview Jenny Lee from XO Group (the company behind The Knot, a major wedding website in America) discusses the company's journey of adopting, provisioning, and evaluating Apache Mesos and DC/OS in production on AWS. The talk focuses on bridging the gap between deciding to use software and operating it in a production environment, exploring infrastructure provisioning, configuration challenges, custom token authentication pipelines, and the eventual architectural shift away from running general container workloads on Mesos.

### Key Topics - **DC/OS vs. Vanilla Mesos:** Evaluating DC/OS CloudFormation templates and AMIs versus deploying native Mesosphere packages on Debian/Ubuntu systems using infrastructure-as-code tools. - **Infrastructure Provisioning with Terraform:** Managing AWS state, instances, and outputs (such as master and slave IP addresses) reproducibly, allowing simple parameter scaling from 5 to 7 masters. - **Configuration Management with Ansible:** Replacing complex CloudFormation user data shell scripts embedded in JSON with modular Ansible playbooks executing tasks in a guaranteed top-to-bottom order for distinct components (Docker, Chronos, Mesos masters, and slaves). - **Private Docker Registry Authentication:** Solving authentication challenges for Amazon Elastic Container Registry (ECR) across Mesos slave nodes using scheduled Chronos jobs, S3, and IAM roles. - **Container Orchestration Decisions:** Why XO Group ultimately shifted container workloads to Docker Swarm while retaining DC/OS strictly for Elasticsearch.

### Technical Details - **Provisioning Trade-offs:** The default DC/OS CloudFormation template provided only three masters, whereas production resilience required at least five (to maintain quorum after a failure). Additionally, DC/OS AMIs installed every binary on all nodes indiscriminately and embedded roughly the last 10% of critical configuration into CloudFormation data, making maintenance, parameterization, and debugging difficult. Small configuration updates required rebuilding entire AMIs. - **Terraform & Ansible Pipeline:** XO Group opted for Terraform to provision AWS infrastructure and track state, using environment variables (`TF_VAR_...`) to supply AWS credentials. Terraform outputs were piped into Ansible inventories. Ansible playbooks cleanly separated installation tasks for Docker, Mesos masters, slaves, and Chronos using Mesosphere's Debian/Ubuntu packages. - **ECR Token Refresh Workflow:** Amazon ECR auth tokens expire every 12 hours. Because slave containerizers require valid tokens to pull private images, two Chronos-managed Docker containers were deployed: 1. The first container runs an AWS CLI command assuming an IAM role to generate an ECR auth token and writes it to an Amazon S3 bucket. 2. The second container pulls the token from S3 and distributes it to slave nodes every six hours, completely avoiding static AWS access keys. - **Architecture Pivot:** XO Group decided against using Mesos/Marathon for general container workloads due to the friction of introducing another interface (Marathon UI/JSON) to developers compared to Docker Swarm (SwarmKit), ZooKeeper's latency sensitivity across availability zones, and DC/OS's proprietary CLI. Instead, they adopted Docker Swarm with Consul for service discovery across regions, while keeping DC/OS solely to run a newer version of Elasticsearch compared to Amazon's managed Elasticsearch service.

### Takeaways - Automating infrastructure with modular tools like Terraform and Ansible provides significantly better control, cleaner state management, and easier debugging than monolithic AMIs and CloudFormation scripts. - Relying on IAM roles and scheduled jobs (such as via Chronos) enables secure, automated token distribution for short-lived cloud credentials without hardcoding static keys. - Developer workflow and operational simplicity often dictate technology choices: even if an orchestrator functions properly, interface complexity, proprietary tooling, and strict latency sensitivities may justify running lightweight alternatives like Docker Swarm for application workloads.