Code RoomTimeout reduction triggers retry storm
MediumPrep Room Coding #2432

Timeout reduction triggers retry storm

On-callCode quality & reviewMid–Senior~30 min

An SRE lowers the HTTP client timeout to a downstream auth service from 2s to 200ms via a config push at 16:40 (intended to fail faster). At 16:42 the auth service's request rate triples, its CPU saturates, and the calling API's success rate drops from 99.9% to 88%. The auth service itself didn't change and its p50 latency is 150ms but p99 is 400ms. The client retries failed auth calls up to 3 times. Triage and mitigate.

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

0:00 of about 30 min
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.