Java service thread pool blocked

On-call incidentHard28 min

Your task

Your Java service runs fine for weeks, then over ~20 minutes goes unresponsive: latency climbs to timeouts, the request thread pool is fully exhausted, but CPU sits at only 25% and memory is normal. Thread dumps show nearly all worker threads BLOCKED inside calls to a downstream recommendation API that recently got slow (its p99 went from 50ms to 8s after their deploy). No autoscaling event helped, new instances exhaust their thread pools the same way. How do you triage and mitigate?

It covers

  • Thread pool exhaustion
  • Blocking call
  • Saturation