Question

Traffic to your GPU-backed LLM inference service rises 3x for a product launch. The HPA is configured to scale on CPU utilization, but CPU stays at ~35% while GPUs are pinned at 100% and p99 latency climbs and requests start timing out — yet no new pods are added. When you manually bump replicas, the new GPU pods sit Pending for several minutes before serving. Dashboards: GPU utilization 100%, GPU memory high, CPU low, cluster GPU node count flat, and a pending-pod count rising. Triage and stabilize.