Connection pool pinned at max
Your task
Your API service calls an upstream profile service through a fixed-size HTTP connection pool (max 50 connections). At 17:00 the profile service had a brief 30s slowdown and is now fully recovered (its p99 back to 40ms). But your API's latency and timeouts haven't recovered, endpoints that call profile still show p99 ~3s and ~8% timeouts 10 minutes later, while endpoints that DON'T call profile are fine. Dashboards: your pool's 'connections in use' is pinned at 50 (100%) and 'wait queue depth' is high; profile-service inbound latency is healthy. How do you triage and mitigate?
It covers
- Connection pool
- Head of line
- Saturation