Code RoomRetry loop after recovery
HardPrep Room Coding #2760

Retry loop after recovery

On-callReliability & on-callSenior–Staff~35 min

A worker fleet's CPU jumps to 95% across all instances at 18:00 and stays there even after the original problem clears. Useful work output (jobs completed/sec) collapses while CPU stays pinned. A profile shows most CPU in a tight retry loop: serialize request, call a dependency, catch the error, immediately retry with no backoff. The trigger was a brief 90-second blip in that dependency at 17:58; the dependency is healthy again, but CPU never recovered. Job arrival rate is flat. Triage and mitigate.

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

0:00 of about 35 min
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.