Retry storm locks in saturation
Your task
A brief 90-second traffic spike pushed your service past capacity. The spike passed minutes ago, traffic is now back to normal levels, but the service has not recovered, it's stuck at ~70% error rate and high latency. Dashboards show CPU pinned at 100%, the request queue full, clients timing out and retrying aggressively (each failed call retries 3x), and goodput near zero even though arriving original load would fit comfortably under capacity. Restarting a few instances briefly helps, then they re-saturate. There was no deploy. How do you triage, and how do you break the service out of this state?
It covers
- Metastable failure
- Load shedding
- Retry amplification