Message loss on eviction
A `sms-dispatch` worker consumes a RabbitMQ queue and sends SMS via a carrier API. After a pod eviction storm during a node drain at 21:00, reconciliation shows ~1,800 messages that were *consumed* (no longer in the queue) but never resulted in an SMS — silent loss. No DLQ, no requeue, no consumer errors logged for those. Dashboards: during the drain, several worker pods were SIGKILLed. The consumer was set up with `autoAck=true` (RabbitMQ `no_ack`) and a `prefetch` of 200 to maximize throughput. How do you triage and mitigate?
What a strong answer looks like
Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.
0:00 of about 40 min
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.
Run or narrate your approach, then ask the coach.