Code Room
On-callHardoc-g390
Subject Message lossLevel Senior–Staff~40 minCommon in Reliability & on-call interviewsIndustries Technology

Question

A `sms-dispatch` worker consumes a RabbitMQ queue and sends SMS via a carrier API. After a pod eviction storm during a node drain at 21:00, reconciliation shows ~1,800 messages that were *consumed* (no longer in the queue) but never resulted in an SMS — silent loss. No DLQ, no requeue, no consumer errors logged for those. Dashboards: during the drain, several worker pods were SIGKILLed. The consumer was set up with `autoAck=true` (RabbitMQ `no_ack`) and a `prefetch` of 200 to maximize throughput. How do you triage and mitigate?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Diagram & narrate the incident
Loading whiteboard…
Run or narrate your approach, then ask the coach.