On-callHardoc-g108

Subject Traffic surgeLevel Senior–Staff~45 minCommon in Reliability & on-call interviewsIndustries Technology

Question

A brief 2x traffic blip pushes a backend service slightly over capacity for ~30 seconds. Latency rises, clients hit their timeouts and retry, and now the service is receiving far more than 2x — it's stuck at 100% CPU with a huge request queue and serving almost nothing useful, even though the original traffic blip ended minutes ago. Removing the extra load doesn't help; the service stays wedged until you intervene. The dashboards show timeouts → retries → more load in a self-sustaining loop. How do you triage and break the service out of this state?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Learn the concepts

Diagram & narrate the incident

Loading whiteboard…

Run or narrate your approach, then ask the coach.