Code RoomRetry storm cascades downstream
MediumPrep Room Coding #2711

Retry storm cascades downstream

On-callReliability & on-callMid–Senior~35 min

A downstream auth dependency starts returning slow 503s for ~5% of calls (a partial brownout, not down). Within minutes your fleet's outbound request rate to it **quadruples** and the dependency tips fully over, then your own API saturates too — load on the dependency is now far higher than the 5% that originally failed. CPU on your nodes is high but mostly in waiting/retrying. There was no traffic spike from users and no deploy. The dependency owner blames you for 'hammering' them. Who's right, and how do you triage and stop the cascade?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

0:00 of about 35 min
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.