Certificate expiry at scale

On-call incidentMedium30 min

Your task

At 00:00 UTC on the 1st of the month, every call from your services to one internal downstream API starts failing simultaneously and completely, 100% error rate, instant, across all regions at once. The errors are TLS handshake failures: certificate has expired. No deploy went out; nothing changed in your code; it broke at a clean clock boundary. The downstream team is paged too. Triage, mitigate, and describe the prevention angle.

It covers

  • Expired certificate
  • Tls
  • Clock