Thread pool deadlock from nested executor submission

On-call incidentHard40 min

Your task

A Java/Tomcat monolith starts returning 503s and 'connection refused' at the load balancer at 12:30. Tomcat's busy-thread count is pinned at the maxThreads of 200 and the accept queue is full. A thread dump shows ~190 threads in WAITING state, all blocked at the same point: they're inside the request-handling thread pool, having submitted a sub-task to the same thread pool and calling Future.get() waiting for it to complete. CPU is near idle. A deploy at 12:15 'parallelized the report-generation step using the shared executor.' Triage and explain the failure precisely.

It covers

  • Blocking io
  • Deadlock
  • Tomcat