On-callMediumoc-g452

Subject Error rate spikesLevel Mid–Senior~35 minCommon in Reliability & on-call interviewsIndustries Technology, Software development

Question

After a routine HTTP-client library upgrade, your service starts seeing a small but steady ~2% rate of connection-reset and 'connection refused' errors to a downstream API, plus a latency bump. The errors climb with traffic. The downstream is healthy and underloaded and reports nothing wrong. CPU/mem are fine. Comparing connection metrics before/after the upgrade, the count of new TCP connections + TLS handshakes per second jumped ~30x, and the average requests-per-connection dropped from ~200 to ~1. There were no other changes. How do you triage and fix?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Learn the concepts

Diagram & narrate the incident

Loading whiteboard…

Run or narrate your approach, then ask the coach.