Product pages fail while the deploy keeps rolling out

On-call incidentEasy15 min

Your task

It is 14:46. The catalog API runs on Amazon ECS behind an Application Load Balancer. Deploys go through CodeDeploy, which moves 10% of traffic to the new version every 5 minutes. It is set to stop and roll back if the CloudWatch alarm catalog-5xx goes into the ALARM state. Support pages you:

SupportEscalation: 22 tickets tagged "product page" in 15 minutes (threshold 10)

What you see:

  • Customers see "Something went wrong" on some product pages. Traffic began moving to the new version at 14:20, and 60% of it reaches the new version now.
  • The load balancer's HTTPCode_Target_5XX_Count, graphed as a sum per minute, rose from about 4 before 14:20 to about 2,200 now. It climbs at each traffic step.
  • catalog-5xx is in OK, as it has been since it was created in June. Its settings: metric HTTPCode_Target_5XX_Count, statistic Average, period 1 minute, above 50 for 2 of 2 periods.
  • The alarm's own graph is a flat line at 1.

What do you do in the next 5 minutes, and what must be true before this service deploys again?

It covers

  • Cloudwatch alarms
  • Deploy safety
  • Load balancer metrics