Code RoomRetry loop keeps service wedged
HardPrep Room Coding #2487

Retry loop keeps service wedged

On-callReliability & on-callSenior–Staff~45 min

A brief 2x traffic blip pushes a backend service slightly over capacity for ~30 seconds. Latency rises, clients hit their timeouts and retry, and now the service is receiving far more than 2x — it's stuck at 100% CPU with a huge request queue and serving almost nothing useful, even though the original traffic blip ended minutes ago. Removing the extra load doesn't help; the service stays wedged until you intervene. The dashboards show timeouts → retries → more load in a self-sustaining loop. How do you triage and break the service out of this state?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

0:00 of about 45 min
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.