Secrets distribution at scale

DesignHard45 min

Your task

Design a secrets-distribution system for ~8,000 microservice instances that need database credentials, API keys, and TLS certs. Requirements: no long-lived secret ever sits in an env var or image; credentials rotate automatically; a leaked credential is useless within minutes; and a full audit trail of who/what accessed which secret. Static-secret reads peak at 30K/sec. Discuss dynamic (short-lived, on-demand) secrets vs rotating static secrets, how a service authenticates to the secrets service in the first place (the bootstrapping/'secret zero' problem), and the trade-off when the secrets service is down.

It covers

  • Dynamic secrets
  • Rotation
  • Lease