Delivery orders crawl as the dinner rush starts

On-call incidentEasy15 min

Your task

It is 19:12 in New York, where your team and your customers are. New York is 4 hours behind UTC this month. The food delivery API runs on an EC2 Auto Scaling group. A target tracking policy keeps average CPU near 50%, between 4 and 24 instances. This alert fires:

OrderApiLatencyP95: p95 above 1.5 s for 5 minutes (current 7.8 s)

What you see:

  • p95, the time 95% of requests finish within, jumped at 19:01. Dinner orders run from 18:00 to 21:00, and tonight's order rate matches last Tuesday's.
  • The group had 16 instances at 18:59. At 19:00 it dropped to 2. At 19:04 it grew to 4, and it has stayed there with CPU at 98%.
  • The group's activity history has: At 2026-10-20T23:00:00Z a scheduled action update of AutoScalingGroup constraints to min: 2, max: 4, desired: 2 changing the desired capacity from 16 to 2.
  • Yesterday a pull request titled "Cut overnight cost" added 2 scheduled actions. 0 23 * * * sets min 2, max 4 and desired 2. 0 7 * * * sets min 4, max 24 and desired 8. Its description says: "Orders are near zero from 23:00 to 07:00."

How do you get through tonight's dinner rush, and what must change before 19:00 tomorrow?

It covers

  • Scheduled scaling
  • Time zones
  • Ec2 auto scaling