SSL Hell

The talk starts with the cost of managing TLS certificates by hand across many microservices. The speakers, who work on an identity management product on AWS and GCP, describe staging outages caused by misconfigured or expired certificates. They recall an April outage at Epic Games, where one expired wildcard certificate deployed across hundreds of backend servers caused a multi-hour disruption, and they note that manual tracking often still relies on spreadsheets.

The middle section introduces ACME as a protocol for automated issuance and renewal. A client requests a certificate and proves control of a domain through an HTTP or DNS challenge before the certificate authority signs it. A live demo runs a command-line ACME client against an nginx server using Let's Encrypt, followed by manual HTTP and DNS challenge flows.

The final part covers cert-manager, a Kubernetes add-on that applies this process across services. Issuers, either namespace-scoped or cluster-wide, define the ACME provider and DNS solver, while certificate resources list domains and a renewal window, such as 15 days before expiry. Results land in Kubernetes secrets, optionally as Java keystores or PKCS#12 files, and ingress controllers pick them up. The speakers flag limitations: it only runs on Kubernetes, private keys sit in secrets that need protection, the controller itself must be monitored after a renewal bug they hit, and Let's Encrypt enforces per-domain issuance limits that separate environments under one domain can exhaust quickly.