GPU autoscaler CPU metric mismatch

On-call incidentHard35 minOpenAI and Nvidia style

Your task

Traffic to your GPU-backed LLM inference service rises 3x for a product launch. The HPA is configured to scale on CPU utilization, but CPU stays at ~35% while GPUs are pinned at 100% and p99 latency climbs and requests start timing out, yet no new pods are added. When you manually bump replicas, the new GPU pods sit Pending for several minutes before serving. Dashboards: GPU utilization 100%, GPU memory high, CPU low, cluster GPU node count flat, and a pending-pod count rising. Triage and stabilize.

It covers

  • Autoscaler
  • Gpu
  • Scaling
  • Cold start