Orders API returns 503s

On-call incidentMedium30 min

Your task

The Java orders API (Tomcat, 200-thread pool) starts returning 503s and the load balancer marks half the fleet unhealthy. Dashboards: active-threads gauge pinned at 200 on every pod, request queue depth climbing, but app CPU is only 25% and GC is calm. The threads dump shows ~190 threads parked in SocketRead0 on calls to the third-party tax-calculation API. That tax vendor posted a status-page notice 10 minutes ago: 'elevated latency.' No deploy today. How do you respond?

It covers

  • Blocking call
  • Saturation
  • Downstream slowness