On-callHardoc-g306

Subject Upstream timeoutLevel Senior–Staff~40 minCommon in Distributed systems interviewsIndustries Technology

Question

Your service holds a pool of long-lived keepalive HTTP/2 connections to an upstream behind a load balancer. At 02:30 the upstream fleet was scaled in (some instances removed) by an autoscaler. Right after, your service starts seeing ~5% of requests fail with connection resets / 'broken pipe' / timeouts, and it slowly self-heals over ~10 minutes. Dashboards: the upstream is healthy and underloaded; the failing requests all reuse existing pooled connections; brand-new connections always succeed. No deploy on your side. How do you triage and mitigate?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Learn the concepts

Diagram & narrate the incident

Loading whiteboard…

Run or narrate your approach, then ask the coach.