CPU throttling under load

On-call incidentMedium30 min

Your task

An image-thumbnailing service (Go) runs on Kubernetes. After a routine traffic increase (a marketing push doubled uploads), p99 latency degraded badly and requests started timing out, but per-pod CPU utilization shown in your dashboard reads only ~55%, nowhere near 100%. The HPA isn't scaling up because it's targeting 70% CPU and 55 < 70. The pod has resources.limits.cpu: 1 and requests.cpu: 0.5. The container metric container_cpu_cfs_throttled_periods is high and rising. Triage and mitigate.

It covers

  • Cpu throttling
  • Kubernetes
  • Limits