Message retry loop exhausts broker

On-call incidentHard40 min

Your task

An order-fulfillment service uses RabbitMQ with a delayed-retry design: the fulfillment work queue dead-letters (via DLX) to fulfillment.wait which has a per-message TTL of 60s and, on TTL expiry, dead-letters back to fulfillment to be retried. There is no explicit attempt counter. At 09:10 broker CPU spikes to ~90%, network throughput triples, and the fulfillment.wait queue depth oscillates around 80k while fulfillment ready-count flickers. The main consumers are healthy and acking normally; effective useful throughput has actually dropped. Recent context: a config push 30 min ago made the inventory gRPC call time out for a specific SKU family that's ~5% of orders. How do you triage and mitigate?

It covers

  • Retry loops back
  • No attempt cap
  • Amplification