Upstream scale-in closes pooled connections

On-call incidentHard40 min

Your task

Your service holds a pool of long-lived keepalive HTTP/2 connections to an upstream behind a load balancer. At 02:30 the upstream fleet was scaled in (some instances removed) by an autoscaler. Right after, your service starts seeing ~5% of requests fail with connection resets / 'broken pipe' / timeouts, and it slowly self-heals over ~10 minutes. Dashboards: the upstream is healthy and underloaded; the failing requests all reuse existing pooled connections; brand-new connections always succeed. No deploy on your side. How do you triage and mitigate?

It covers

  • Keepalive
  • Stale connection
  • Half open