Practicing Cloud-Native Resliency Is a Challenge: Decoding It with Chaos...

Praj, community manager at Harness and CNCF Ambassador for Litmus Chaos, presents chaos engineering as an essential practice for cloud-native systems resilience. The talk addresses why chaos engineering matters: system complexity, rapid deployment cycles (100x faster than monoliths), and multi-layered failure possibilities. Downtime costs extend beyond financial loss to include developer burnout and lost productivity—a single engineer at $100k salary losing time to debugging represents substantial organizational cost.

Chaos engineering is framed not as a luxury but as an integration testing framework where teams deliberately induce faults to identify weaknesses proactively, before production incidents. The methodology follows a structured process: identify steady state, select scenarios/experiments, inject faults in controlled manner, observe metrics and logs, return to steady state, and use findings for resilience improvements. This "chaos first principle" was pioneered by Netflix's Adrian Cockroft in 2012–2013.

Litmus Chaos, a CNCF incubating project, provides the tooling. With 500+ companies using it and 30 million Docker pulls, Litmus features: Control Plane (authentication, MongoDB, GraphQL UI), Execution Plane (Argo CD workflows, custom resource operators), and a Chaos Hub marketplace (~50 predefined experiments). Experiments cover Kubernetes, AWS, Azure, GCP, Spring Boot, Kafka, and Cassandra. Mandatory probes (health checks) in Litmus 3.0 verify system recovery via HTTP or other modes.

Game days—structured chaos events—require defining stakeholders, environments, workloads, observability tools (Datadog, Dynatrace, Grafana), guard rails, rollback mechanisms, and alerts. The demo showed pod deletion against a microservices app (online boutique), targeting the cart service to measure resilience.

Real-world adoption spans industries: Delivery Hero (4,000+ services with circuit breaking), Adidas (staging to production progression), FIS (load generation + chaos injection with Dynatrace monitoring and Keptn SLO evaluation), and Flipkart (VM chaos engineering). Adoption patterns show teams starting with SRE/QA, then expanding to multiple teams and developers.

Starting with chaos requires identifying stakeholders, selecting tools, and running small game days in pre-production before automation and scaling. The project roadmap includes maturity models, SLSA compliance, threat modeling, security frameworks, and open-source budgeting guidance. Litmus aims for CNCF graduation, with community contributions invited for code, experiments, documentation, and website improvements.