New product photos show placeholders instead of thumbnails

On-call incidentEasy15 min

Your task

It is 14:10. When sellers upload product photos, the image service puts a resize job on an SQS queue. Workers in an EC2 Auto Scaling group take jobs from it. The group runs only Spot Instances: spare EC2 capacity sold at a discount, which AWS can take back with 2 minutes' notice. All of them are c5.2xlarge, spread over 2 Availability Zones. At the afternoon peak the group runs 20 instances. This alert fires:

ResizeQueueAge: oldest job is 16 minutes old (threshold 10 minutes)

What you see:

  • At 13:41, 12 of the 20 instances got a Spot interruption notice, and they stopped at 13:43.
  • Since 13:43 the group's activity history repeats, every few minutes: Could not launch Spot Instances. InsufficientInstanceCapacity - There is no Spot capacity available that matches your request. Launching EC2 instance failed.
  • Each instance finishes about 75 jobs a minute. At this hour sellers add about 1,400 jobs a minute. The queue holds about 22,000 jobs and grows by about 800 a minute.
  • New listings show a grey placeholder until their thumbnails are ready. Sellers have opened 35 tickets.

How do you clear the backlog this afternoon, and how do you change the group so this cannot stall the queue again?

It covers

  • Spot instances
  • Instance diversity
  • Queue backlog