Message retry storm collapses throughput
Your task
At 10:40 your SQS-backed worker fleet's processing throughput collapses and the queue depth explodes to millions. Dashboards: a single message type started failing at 10:35; the consumers retry failed messages, and because visibility-timeout redelivery + your app-level retry both fire, the same messages cycle endlessly; CPU on workers is high but 'useful work done' (successful completions) is near zero; the DLQ is empty (maxReceiveCount is set very high). A schema change shipped at 10:34. How do you triage and mitigate?
It covers
- Queue backup
- Dlq
- Poison message