Message loss on eviction
Your task
A sms-dispatch worker consumes a RabbitMQ queue and sends SMS via a carrier API. After a pod eviction storm during a node drain at 21:00, reconciliation shows ~1,800 messages that were consumed (no longer in the queue) but never resulted in an SMS, silent loss. No DLQ, no requeue, no consumer errors logged for those. Dashboards: during the drain, several worker pods were SIGKILLed. The consumer was set up with autoAck=true (RabbitMQ no_ack) and a prefetch of 200 to maximize throughput. How do you triage and mitigate?
It covers
- Auto ack
- Crash window
- Prefetch loss