Thread pool deadlock under load
Your task
v33 of a Java service deploys at 08:30. Under normal load it works, but during the morning traffic ramp at 09:15 it locks up: requests hang indefinitely, the service's health check still returns 200, but no requests complete and CPU drops to near zero. A thread dump shows all threads in a fixed-size thread pool BLOCKED, and many are waiting to submit work to that same pool. v33 introduced a pattern where a request task, running on the pool, synchronously submits a sub-task to the same bounded pool and waits for its result. Triage and mitigate.
It covers
- Thread pool
- Blocking io
- Deadlock