Blocking call starves event loop

On-call incidentHard35 min

Your task

A reactive (Project Reactor / Netty) Java service that normally handles 50k RPS on a handful of event-loop threads suddenly drops to handling almost nothing, latency goes to seconds, throughput collapses, but CPU is near idle (15%). Thread dumps show all of the small number of event-loop threads BLOCKED, parked inside a synchronous JDBC call and a .block() added in last night's deploy to 'quickly reuse' a legacy DAO. No upstream/downstream change. How do you triage and mitigate?

It covers

  • Thread pool exhaustion
  • Blocking in async
  • Event loop