Retry storm cascades downstream
Your task
A downstream auth dependency starts returning slow 503s for ~5% of calls (a partial brownout, not down). Within minutes your fleet's outbound request rate to it quadruples and the dependency tips fully over, then your own API saturates too, load on the dependency is now far higher than the 5% that originally failed. CPU on your nodes is high but mostly in waiting/retrying. There was no traffic spike from users and no deploy. The dependency owner blames you for 'hammering' them. Who's right, and how do you triage and stop the cascade?
It covers
- Retry storm
- Self inflicted
- Thundering herd
- Backpressure