Outbound connections fail under load

On-call incidentHard40 min

Your task

A backend service that fans out to a downstream API starts intermittently failing to make outbound connections under peak load: logs show 'cannot assign requested address' (EADDRNOTAVAIL) on connect, and request error rate spikes during traffic peaks then recovers in troughs. The host is not CPU- or memory-bound. ss -s shows tens of thousands of sockets in TIME_WAIT toward the downstream's IP:port. A recent change replaced a pooled/keep-alive HTTP client with one that opens a fresh connection per request. The ephemeral port range is the default. Triage and fix.

It covers

  • Ephemeral ports
  • Keep alive
  • Port exhaustion