Connection reuse regression

On-call incidentMedium35 min

Your task

After a routine HTTP-client library upgrade, your service starts seeing a small but steady ~2% rate of connection-reset and 'connection refused' errors to a downstream API, plus a latency bump. The errors climb with traffic. The downstream is healthy and underloaded and reports nothing wrong. CPU/mem are fine. Comparing connection metrics before/after the upgrade, the count of new TCP connections + TLS handshakes per second jumped ~30x, and the average requests-per-connection dropped from ~200 to ~1. There were no other changes. How do you triage and fix?

It covers

  • Error rate spikes
  • Connection churn
  • Keepalive