Account vCPU quota exceeded

On-call incidentMedium30 minAmazon style

Your task

A routine autoscaling event during evening peak fails: new EC2 instances launch but stay pending then terminate, and your ASG activity log shows Could not launch On-Demand Instances. VcpuLimitExceeded for your instance family in us-east-1. Meanwhile pods elsewhere on EKS are stuck ContainerCreating with failed to assign an IP address to container in the kubelet logs. Traffic is up 40% over a normal Tuesday because of a marketing email that went out at 18:00. How do you triage this and restore capacity?

It covers

  • Cloud quota
  • Ip addresses
  • Deploy