Client retry amplifies traffic surge

On-call incidentHard30 minGoogle and Netflix style

Your task

A mobile-app release this week added an aggressive client-side retry to a flaky-feeling endpoint. Today a genuine 2x organic surge hits and your backend, which has handled larger real surges before, falls over: it's pinned at 100% CPU, the load balancer shows incoming request rate ~7x baseline (far above the 2x of real users), and a chunk of that is duplicate requests carrying the same idempotency key seconds apart. Pulling the app open-rate, actual user activity is only up ~2x. How do you triage and mitigate?

It covers

  • Retry budget
  • Client amplification
  • Deploy