HPA oscillates near target threshold

On-call incidentHard30 min

Your task

Your recommendations service runs on a Kubernetes HPA targeting 65% CPU, min 8 / max 40 pods. Since this morning the replica count oscillates wildly, scaling 8 to 36 to 8 to 34 every few minutes, and p99 latency has a sawtooth pattern that tracks the swings. Each scale-down evicts warm pods, and the surviving pods briefly spike to 95% CPU before new ones become ready (your readiness probe gates on a 40s warm-cache load). Dashboards show average CPU hovering right at the 65% target and the HPA event log is full of alternating SuccessfulRescale up/down events. Traffic is flat versus last week. Walk through your triage and how you'd stop the flapping.

It covers

  • Hpa thrash
  • Flapping
  • Scale oscillation