Order service latency spikes

On-call incidentHard40 min

Your task

Your order service calls a downstream inventory service that is slow but NOT down, its p99 crept from 60ms to 1.2s at 14:00 and has held there. Within ten minutes your order service, which makes many calls that don't even touch inventory, is failing broadly: p99 to 8s, 30% errors on endpoints that have nothing to do with inventory. Dashboards: inventory itself is at a steady 1.2s, not getting worse; your order service's CPU is low; its outbound connection pool to inventory is pegged at 100% utilization with a long wait queue. No deploy. How do you triage and mitigate?

It covers

  • Connection pool
  • Slow not down
  • Head of line