GPU autoscaler CPU metric mismatch
Your task
Traffic to your GPU-backed LLM inference service rises 3x for a product launch. The HPA is configured to scale on CPU utilization, but CPU stays at ~35% while GPUs are pinned at 100% and p99 latency climbs and requests start timing out, yet no new pods are added. When you manually bump replicas, the new GPU pods sit Pending for several minutes before serving. Dashboards: GPU utilization 100%, GPU memory high, CPU low, cluster GPU node count flat, and a pending-pod count rising. Triage and stabilize.
It covers
- Autoscaler
- Gpu
- Scaling
- Cold start