Delivery orders crawl as the dinner rush starts
Your task
It is 19:12 in New York, where your team and your customers are. New York is 4 hours behind UTC this month. The food delivery API runs on an EC2 Auto Scaling group. A target tracking policy keeps average CPU near 50%, between 4 and 24 instances. This alert fires:
OrderApiLatencyP95: p95 above 1.5 s for 5 minutes (current 7.8 s)
What you see:
- p95, the time 95% of requests finish within, jumped at 19:01. Dinner orders run from 18:00 to 21:00, and tonight's order rate matches last Tuesday's.
- The group had 16 instances at 18:59. At 19:00 it dropped to 2. At 19:04 it grew to 4, and it has stayed there with CPU at 98%.
- The group's activity history has:
At 2026-10-20T23:00:00Z a scheduled action update of AutoScalingGroup constraints to min: 2, max: 4, desired: 2 changing the desired capacity from 16 to 2. - Yesterday a pull request titled "Cut overnight cost" added 2 scheduled actions.
0 23 * * *sets min 2, max 4 and desired 2.0 7 * * *sets min 4, max 24 and desired 8. Its description says: "Orders are near zero from 23:00 to 07:00."
How do you get through tonight's dinner rush, and what must change before 19:00 tomorrow?
It covers
- Scheduled scaling
- Time zones
- Ec2 auto scaling