Retry storm after brief outage

On-call incidentHard40 minGoogle style

Your task

A brief 5-second blip in a downstream auth service (now fully recovered) has left your whole API tier wedged: error rate is stuck at 60%, the auth service is pegged at 100% CPU even though the underlying fault is gone, and traffic into your tier is normal. Restarting individual API pods doesn't help, they re-wedge within seconds. Each client SDK retries failed auth calls 3x with fixed 200ms backoff, and your API layer also retries auth 2x on its own. The system won't recover on its own even though nothing is currently broken. How do you triage and break the loop?

It covers

  • Retry storm
  • Metastable failure
  • Backoff