Thread pool exhaustion from slow dependency

On-call incidentMedium35 min

Your task

Your Java service that aggregates data from three downstream APIs starts returning 503s and RejectedExecutionException under normal traffic. The dashboards show the request thread pool at 100% busy with a long queue, but service CPU is only 45% and GC looks healthy. One of the three downstream APIs (the 'pricing' service) has p99 quietly climbed from 50ms to 2.5s over the last 20 minutes. Your service calls all three downstreams synchronously on the same thread pool. How do you triage and stop your service from falling over?

It covers

  • Thread pool
  • Downstream latency
  • Jvm