Retry loop keeps service wedged
Your task
A brief 2x traffic blip pushes a backend service slightly over capacity for ~30 seconds. Latency rises, clients hit their timeouts and retry, and now the service is receiving far more than 2x, it's stuck at 100% CPU with a huge request queue and serving almost nothing useful, even though the original traffic blip ended minutes ago. Removing the extra load doesn't help; the service stays wedged until you intervene. The dashboards show timeouts to retries to more load in a self-sustaining loop. How do you triage and break the service out of this state?
It covers
- Retry amplification
- Metastable
- Overload