Queue throttling synchronous path

On-call incidentMedium30 minAmazon style

Your task

Your order pipeline enqueues every order event into a managed message queue (SQS-style). At 15:00 your order-API p99 jumps from 120ms to 4s and ~10% of orders fail. Dashboards: SendMessage calls to the queue are returning throttling errors ('RequestThrottled' / 429) and high latency; the queue's API request rate hit a soft account limit after a promo doubled order volume. Your API enqueues synchronously inside the request path and retries the send up to 3 times before failing the order. Your own service CPU/DB are healthy. How do you triage and mitigate?

It covers

  • Managed queue
  • Throttling
  • Backpressure