On-callHardoc-g323

Subject Capacity incidentsLevel Senior–Staff~28 minCommon in Concurrency · Reliability & on-call interviewsIndustries Technology

Question

Your Java service runs fine for weeks, then over ~20 minutes goes unresponsive: latency climbs to timeouts, the request thread pool is fully exhausted, but CPU sits at only 25% and memory is normal. Thread dumps show nearly all worker threads BLOCKED inside calls to a downstream recommendation API that recently got slow (its p99 went from 50ms to 8s after their deploy). No autoscaling event helped — new instances exhaust their thread pools the same way. How do you triage and mitigate?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Learn the concepts

Diagram & narrate the incident

Loading whiteboard…

Run or narrate your approach, then ask the coach.