Zero downtime credential rotation

The talk starts from a familiar problem: teams avoid changing service credentials because they fear breaking systems or causing costly outages. The speaker argues that credentials will eventually leak, so they should be rotated quickly and routinely, and that practicing rotation reveals hidden weaknesses in a system before an incident does.

He separates symmetric credentials, such as passwords and API keys, from derived, short-lived ones like JSON web tokens and certificates. He contrasts push models, where secrets are injected at deploy time, with pull models, where applications re-read secrets as needed. Three rotation approaches are presented: overlapping blue and green credentials, fail-and-retry against the secret manager, and refreshing derived credentials after a fraction of their lifetime has passed. He then describes problems his team hit while rolling this out, including credentials shared across unrelated clients, retry loops without delay that overwhelmed their own services, leaked threads from failed connections, service accounts that could be locked out by anyone who knew the username, and commercial software that needed restarts to pick up new secrets.

The takeaways are to split credentials along logical boundaries, add jitter and backoff, make client libraries tunable with callbacks and file watches, log credential failures at warning level or above, and use layered alerts on both application errors and secret manager access. He also urges teams to rotate during business hours and to publish dashboards so other groups can see recent changes.