DevOpsDays Warsaw 2019 - Ara Pulido

### Overview In this DevOpsDays Warsaw 2019 presentation, speaker Ara Pulido discusses how Datadog manages Kubernetes in production at high scale compared to standard smaller-scale deployments. Prior to joining Datadog, Pulido worked at Bitnami running clusters of 15–25 nodes, whereas Datadog operates dozens of clusters across multiple clouds, each exceeding 1,000 nodes. The talk examines key networking challenges encountered when scaling Kubernetes, focusing on pod networking, service load balancing, and DNS resolution.

### Key Topics - **Scale Differences in Kubernetes**: While survey data indicated that a majority of users run small clusters (1–10 nodes) and only about 4% run more than 100 nodes, operating massive clusters exposes edge cases and performance bottlenecks not seen in smaller setups. - **Pod Networking & CNI Plugins**: Comparison between network overlays (e.g., Flannel, Weave Net), BGP-based routing (e.g., Calico), and direct cloud VPC pod routing (e.g., AWS CNI/EKS plugin, Lyft's plugin). - **Service Networking & kube-proxy**: The limitations of standard `iptables` mode versus IPVS mode for cluster service routing at scale. - **Cluster DNS Resolution**: Addressing known DNS latency issues and race conditions using local DNS caching mechanisms.

### Technical Details - **Pod Networking**: The Container Network Interface (CNI) specification governs pod networking rules: every pod gets a unique IP and can route traffic to any other pod. Basic overlay setups like Flannel wrap packets using IP-in-IP, which incurs CPU overhead and increases packet size at scale. Calico distributes routing tables via BGP without a Docker bridge. In cloud environments, Datadog avoids overlay overhead by employing direct pod routing (such as the AWS VPC CNI plugin or Lyft's plugin), assigning every pod an IP directly within the VPC. - **Service Networking (kube-proxy)**: Default `iptables` mode creates a NAT table that grows linearly with pods and services, requiring a full resync on every update. Datadog uses IPVS mode (GA since Kubernetes 1.11), which provides atomic updates, supports configurable load balancing algorithms (e.g., round-robin, least connection), and maintains stable performance. Calico benchmarks highlighted significant CPU and latency improvements using IPVS starting around 1,000 services. Cilium is also noted as an emerging eBPF-based alternative. - **DNS Handling**: Kubernetes clusters at scale can suffer from a 5-second DNS delay bug due to race conditions during CoreDNS lookups. Datadog addressed this by deploying a custom DNS cache on every node to resolve queries locally before hitting CoreDNS. Kubernetes later introduced this pattern upstream as `node-local-dns` cache (beta in 1.15).

### Takeaways - Kubernetes provides significant flexibility for developers, but maintaining it at scale presents notable operational complexity. - Operations teams must become subject matter experts in their specific CNI plugin, as many scale-related bugs originate in the CNI implementation rather than Kubernetes core. - The open-source cloud native ecosystem evolves rapidly; features like IPVS service routing and node-local DNS caching quickly transitioned from custom operational fixes to upstream Kubernetes features. - Underlying networking changes (CNI plugins, kube-proxy modes, DNS caching) can be introduced incrementally across clusters without impacting developer workflow or requiring large application refactors.