Lambda throttle backlog

On-call incidentMedium30 minAmazon style

Your task

Your payment-webhook fan-out runs SQS to Lambda. At 18:50 the ApproximateNumberOfMessagesVisible on the webhook-dispatch queue jumps from ~500 to 220k and keeps climbing; ApproximateAgeOfOldestMessage is now 9 minutes and rising. The Lambda's ConcurrentExecutions is pinned flat at 1,000 and Throttles is non-zero and growing. A marketing push went out at 18:45 that 3x'd inbound webhook volume. Downstream the merchant-notification API (called by the Lambda) shows p99 latency up from 80ms to 1,100ms. Walk through triage and mitigation.

It covers

  • Sqs
  • Lambda
  • Throttling