Kubernetes Security 101: Best Practices to Secure Your Cluster

### Overview In this introductory presentation, Magno Logan covers the fundamentals of Kubernetes architecture, highlights critical security concerns, and details best practices for securing clusters. After outlining the core components of both the control plane and worker nodes, the talk dives into common security pitfalls—such as exposed APIs, flat networking, unencrypted data stores, and disabled audit logging—before introducing a recommended three-pillar defense model.

### Key Topics - **Cluster Architecture**: Breakdown of control plane components (kube-apiserver, etcd, kube-scheduler, kube-controller-manager, cloud-controller-manager) versus worker node components (kubelet, kube-proxy, runtime engines such as containerd, CRI-O, or Podman). - **Kubernetes APIs and Exposure**: Overview of the three primary APIs (kube-apiserver, etcd API, and kubelet API) and the risks of exposing them to untrusted networks or leaving them unauthenticated. - **Namespaces and Isolation**: Explanation that namespaces act like organizational folders rather than hard security boundaries, meaning compromised pods can interact across namespaces unless restricted by Role-Based Access Control (RBAC). - **Flat Networking & Network Policies**: Addressing the default behavior where every pod can talk to every other pod, and the necessity of enforcing Network Policies with supporting Container Network Interfaces (CNIs). - **Data Protection & Integrity**: Encrypting etcd at rest, verifying file integrity based on CIS Benchmarks, and configuring audit logs for incident response. - **The Kubernetes Security Triad**: A foundational defense strategy comprising image scanning, admission control, and runtime security.

### Technical Details - **API Attack Surfaces**: While `kube-apiserver` acts as the main REST API interface and is frequently exposed, endpoints like `/version` leak Go and Kubernetes release versions. Furthermore, etcd and the kubelet maintain their own APIs. Logan notes real-world exploitation by the threat actor group Team TNT, who scanned internal networks for the kubelet API (port 10250), accessed unauthenticated endpoints, and modified pods across roughly 50,000 compromised environments to mine Monero cryptocurrency. - **etcd Security**: While communication in transit is encrypted by default with TLS certificates, data at rest is not encrypted by default on unmanaged clusters. Tampering with etcd directly alters the cluster's desired state, potentially pulling malicious container images from registries like Docker Hub. - **Network Policies & CNIs**: Mitigating flat networking requires Network Policies to enforce least privilege communication (e.g., preventing frontend pods from directly accessing databases). Logan notes that operators must verify that their chosen CNI (e.g., Calico, Flannel) actually supports Network Policy enforcement. - **File Integrity Monitoring (FIM)**: Applying CIS Kubernetes Benchmarks to track hash changes, file permissions, and ownership on sensitive control plane and node configuration files. - **Audit Logging**: Audit logs are disabled by default in vanilla Kubernetes, though cloud providers like AWS EKS allow one-click activation sending logs to S3 and CloudWatch. Log rotation and filtering are necessary due to the high volume of events generated. - **Triad Tooling**: Admission controllers (formerly PodSecurityPolicy, now Pod Security Admission, as well as Open Policy Agent/OPA or Kyverno) serve as cluster gatekeepers. Runtime security tools like Falco utilize eBPF and audit logs to detect in-container threats—such as unauthorized curl/wget executions or cryptomining processes—often visualized using Falco Sidekick.

### Takeaways - **Assess Complexity First**: Logan advises organizations to evaluate whether they truly need self-hosted Kubernetes due to its maintenance overhead, suggesting managed offerings (EKS, AKS, GKE) as a simpler alternative. - **Reduce the Blast Radius**: Restrict cluster APIs using API gateways, firewalls, and private networking; enforce Network Policies to segment pod traffic. - **Adopt Defense-in-Depth**: Secure the entire lifecycle by scanning images before deployment, strictly gating workloads using admission controllers, and monitoring running containers via eBPF-driven runtime security.