Poison message blocks order queue

On-call incidentMedium35 minAmazon style

Your task

An order-state worker consumes an SQS FIFO queue, partitioned by MessageGroupId = order_id so each order's events process in order. At 10:15 a subset of orders stop advancing, their later events never process, while most orders are fine. Dashboards: ApproximateNumberOfMessagesVisible is flat-ish, but a few message groups show messages stuck and the worker logs an unhandled exception on one message that keeps reappearing after the visibility timeout. No DLQ is configured. How do you triage, unblock the stuck orders, and harden the consumer?

It covers

  • Sqs fifo
  • Message group
  • Redrive
  • Ordered blocking