RabbitMQ retry loop amplifies failures

On-call incidentHard40 minStripe style

Your task

RabbitMQ setup: a payments work queue dead-letters (via DLX) to a payments-retry queue with a message TTL of 60s, and that retry queue dead-letters back to payments for another attempt (a delayed-retry pattern). At 08:00, broker CPU and message rates spike: the payments-retry queue depth oscillates around 80k, the payments queue shows a huge redelivered rate, and the broker's overall publish+deliver throughput is ~10x normal with no increase in real business traffic. Recent context: a downstream payment processor started returning errors at 07:45 for ~30% of charges. Triage and explain why the broker is melting, and how you mitigate.

It covers

  • Rabbitmq
  • Dlx
  • Ttl loop