Connection pool saturation

On-call incidentHard40 min

Your task

Your monolith talks to 6 microservices through a shared HTTP client connection pool (max 200). At 19:40 the whole app's p99 goes to 30s and error rate spikes across ALL endpoints, even ones unrelated to search. Dashboards: the recommendations service (one of the 6) has p99 of 28s after a deploy at 19:35; the shared pool shows 200/200 connections in use and a growing wait queue; thread pool on the monolith is saturated. Other 5 downstreams are healthy. How do you triage and mitigate?

It covers

  • Connection pool
  • Bulkhead
  • Timeout