Keynote: Supporting Large-Scale and Reliability Testing in Kubernetes u... Yuan Chen & Shiming Zhang

### Overview In this keynote presentation, Shiming Zhang (Kubernetes SIG Node reviewer and project maintainer) and Yuan Chen (Principal Engineer at Nvidia) present KWOK (referred to phonetically in the transcript as "Cl" / "Cod"), an open-source toolkit designed for creating and managing large-scale Kubernetes clusters using simulated nodes and workloads. The speakers address the challenges of testing scalability, scheduling, and reliability—particularly in massive, expensive GPU clusters used for artificial intelligence and large language models—without needing thousands of physical nodes or real containers.

### Key Topics - **Resource-Efficient Kubernetes Simulation**: Using KWOK controllers to simulate the full lifecycle of nodes, pods, and other Kubernetes API objects natively or via container runtimes on macOS, Linux, and Windows. - **Cluster and Metrics Emulation**: Simulating resource usage metrics (CPU, memory) for the Metrics Server to test Horizontal Pod Autoscaler (HPA), Vertical Pod Autoscaler (VPA), cluster autoscaling, and scheduling policies on a single laptop. - **Reliability and Fault Tolerance in GPU Clusters**: Addressing the high cost of failures in large GPU clusters where failures can propagate across interconnected nodes and disrupt entire AI training workloads. - **Node and Pod Level Error Injection**: Injecting hardware and software failures, custom exit codes, and error conditions into simulated environments to validate proactive and reactive fault-tolerance mechanisms.

### Technical Details - **Controller Architecture**: The simulation controller mimics standard kubelet behaviors and object lifecycles. A cluster of thousands of nodes and tens of thousands of pods can run on a laptop with a minimal memory footprint (under 450 MB with metrics simulation, and under 100 MB without). - **CLI Management (`kwokctl`)**: A command-line utility enables users to spin up clusters in seconds, adjust node and pod counts with scaling commands, inspect simulated objects, and take or restore cluster snapshots to repeat test scenarios consistently. - **Metrics Server Simulation**: Resource usage can be defined and dynamically updated via pod annotations to verify how monitoring and autoscaling systems respond to workload changes. - **Fault Injection Mechanisms**: - *Node Level*: Simulates hardware, system, and driver events (such as those from Node Problem Detector and Nvidia Data Center GPU Manager / DCGM) on fake nodes. - *Pod Level*: Introduces container errors by specifying container names, exit codes, custom error messages, and configurable delay timers (with jitter) via pod annotations. - *GPU Cluster Testing*: Demonstrated creating virtual A100 GPU nodes from production specifications and simulating network failures (e.g., NCCL test communication failures returning exit code 3) to test pre-flight checks and fault-tolerant scheduling.

### Takeaways KWOK provides an accessible, low-overhead testing platform that allows engineers to evaluate complex Kubernetes cluster behaviors, autoscaling algorithms, and scheduling logic without physical hardware. At Nvidia, this testing framework is applied to validate proactive pre-flight checks and reactive rescheduling strategies for AI infrastructure. Future work includes expanding simulation capabilities for GPU operators, next-generation architectures (such as Blackwell/B200), and federating simulations across multi-cluster environments.