Breaking Valkey on Purpose: Chaos Fuzzing a High-Performance Key-Value Store With... Renuka Uttarala

## Overview

Renuka Uttarala from AWS presents a chaos fuzzing approach to test Valkey, an open-source, Linux Foundation-backed in-memory key-value store that serves as a drop-in replacement for Redis 7.2. The talk emphasizes that chaos engineering—intentionally breaking systems—is essential for discovering real-world failure scenarios that unit testing cannot expose.

## Architecture & Key Concepts

Valkey operates in standalone or cluster modes. Cluster mode uses 16,384 hash slots distributed across primary nodes for data distribution, with each primary optionally paired with one or more replicas. Communication occurs via a full mesh gossip protocol (cluster bus) that handles membership discovery, heartbeats, failure detection, and failover coordination.

The speaker demonstrates two primary failure modes found through testing: 1. **Split brain**: Two nodes simultaneously claim primary status for the same slots, causing data divergence. 2. **Stuck failure**: A primary node dies but slots are never reassigned to replicas, resulting in partial outage and quorum loss. 3. **Topology divergence**: Incorrect node ownership when a replica rejoins the cluster as an extra primary.

## Fuzzing Architecture & Testing Methodology

The custom fuzzer orchestrates a complete test lifecycle: - **Cluster Orchestrator**: Configures shards, replicas, replication lag, and base ports - **Chaos Engine**: Injects SIGKILL and SIGTERM signals randomly or on primary nodes - **Validation Layer**: Performs seven checks including slot coverage, data replication consistency, cluster status, and topology correctness - **AI Analysis**: Uses Amazon Bedrock to analyze logs, identify root causes, and suggest new test scenarios

Randomization parameters include 3–16 shards, 1–5 replicas per shard, and configurable YAML-based custom scenarios.

## Demo Results & Real-World Impact

Out of 18 test runs, most passed. One reproducible failure scenario (fixed seed) triggered quorum loss when minimal replicas and aggressive primary killing left insufficient nodes for consensus. The AI agent identified this root cause and suggested increases in replicas per shard and faster failover recovery mechanisms.

Critically, the fuzzer discovered **Issue #85**, a real production bug in Valkey code, which was reported and fixed via pull request—demonstrating the approach's practical value in preventing Saturday morning pages.

## Key Takeaways

1. **Multi-AZ deployment** is mandatory to survive zone failures 2. **Minimum 3+ shards** ensures distributed data and logic 3. **Force failover testing** before production prevents costly incidents 4. **Connection pool drain** improvements needed for chaos scenarios 5. Future testing: network partitions, scale in/out, memory pressure, clock skew

The fuzzer code and daily automated runs are available on GitHub. While the core fuzzer is open-source, the AI agent component (built with AWS Kiro) was created custom but contributors are invited. The speaker notes that AI agents can hallucinate; results must be human-validated and logs larger than context windows should be split into multiple files.