Metrics server down breaks autoscaling

On-call incidentHard40 min

Your task

Your checkout service runs on Kubernetes with an HPA targeting 60% CPU. During a flash sale, latency climbs and pods start OOMKilling, but the HPA holds the deployment at its current 12 replicas and never scales up. kubectl describe hpa shows unknown / 60% for the CPU metric and an event failed to get cpu utilization: unable to fetch metrics from resource metrics API. The metrics-server pod restarted 6 minutes ago. The deployment has maxReplicas: 80 and the cluster has spare nodes. How do you triage and get checkout scaling again?

It covers

  • Hpa
  • Kubernetes
  • Metrics