FIFO queue head-of-line blocking

On-call incidentHard40 minAmazon style

Your task

An SQS FIFO queue inventory-commands (MessageGroupId = warehouse_id) processes per-warehouse commands in order. At 12:40 overall throughput collapses: ApproximateNumberOfMessagesVisible climbs to 70k, oldest-age rises, but the worker fleet is mostly idle, few messages in flight. Dashboards: most active message groups are draining, but one warehouse_id (the new mega-DC onboarded today) has thousands of queued commands. SQS FIFO delivers in-order within a group and won't release the next message in a group until the current one is acked. No errors, no DLQ growth. How do you triage and mitigate?

It covers

  • Fifo throttle
  • Message group skew
  • Inflight limit