Retry storm locks in saturation

On-call incidentHard32 minGoogle and Netflix style

Your task

A brief 90-second traffic spike pushed your service past capacity. The spike passed minutes ago, traffic is now back to normal levels, but the service has not recovered, it's stuck at ~70% error rate and high latency. Dashboards show CPU pinned at 100%, the request queue full, clients timing out and retrying aggressively (each failed call retries 3x), and goodput near zero even though arriving original load would fit comfortably under capacity. Restarting a few instances briefly helps, then they re-saturate. There was no deploy. How do you triage, and how do you break the service out of this state?

It covers

  • Metastable failure
  • Load shedding
  • Retry amplification