Message loss on eviction

On-call incidentHard40 min

Your task

A sms-dispatch worker consumes a RabbitMQ queue and sends SMS via a carrier API. After a pod eviction storm during a node drain at 21:00, reconciliation shows ~1,800 messages that were consumed (no longer in the queue) but never resulted in an SMS, silent loss. No DLQ, no requeue, no consumer errors logged for those. Dashboards: during the drain, several worker pods were SIGKILLed. The consumer was set up with autoAck=true (RabbitMQ no_ack) and a prefetch of 200 to maximize throughput. How do you triage and mitigate?

It covers

  • Auto ack
  • Crash window
  • Prefetch loss