Code RoomUpstream scale-in closes pooled connections
HardPrep Room Coding #2658

Upstream scale-in closes pooled connections

On-callDistributed systemsSenior–Staff~40 min

Your service holds a pool of long-lived keepalive HTTP/2 connections to an upstream behind a load balancer. At 02:30 the upstream fleet was scaled in (some instances removed) by an autoscaler. Right after, your service starts seeing ~5% of requests fail with connection resets / 'broken pipe' / timeouts, and it slowly self-heals over ~10 minutes. Dashboards: the upstream is healthy and underloaded; the failing requests all reuse existing pooled connections; brand-new connections always succeed. No deploy on your side. How do you triage and mitigate?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

0:00 of about 40 min
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.