Autoscaler ignores queue depth

On-call incidentMedium35 minAmazon style

Your task

Your async job system processes webhooks via SQS + a worker fleet that autoscales on CPU. A burst of inbound webhooks (a partner did a backfill) pushes the SQS ApproximateNumberOfMessagesVisible from a few hundred to 1.2M. Queue age climbs to 40 minutes and downstream partners start complaining about delayed callbacks, but the worker autoscaler barely adds capacity, worker CPU sits around 35% because each job spends most of its time waiting on an external HTTP call. How do you triage and clear the backlog?

It covers

  • Queue depth
  • Scaling lag
  • Workers