CPU throttling despite headroom

On-call incidentMedium35 min

Your task

A latency-sensitive service on Kubernetes shows periodic p99 latency spikes to 2s every few seconds even though node CPU utilization sits at a comfortable 40% and the app's own CPU usage looks low. The container has a CPU limit of '500m'. The 'container_cpu_cfs_throttled_periods' and 'throttled_seconds' metrics are high and correlate exactly with the latency spikes. The service is multithreaded (a thread pool sized to the node's many cores). No deploy; traffic is normal. Triage and mitigate.

It covers

  • Throttling
  • Cfs quota
  • Container limits