Control plane requests blocked

On-call incidentHard40 min

Your task

A streaming-upload service (Go) that proxies large client uploads to object storage starts showing badly inflated p99 latency on its small, fast control-plane API endpoints, the lightweight /status and /auth calls that share the same hosts. The big upload streams are healthy and throughput is high; it's the tiny requests that are now slow, with multi-second delays. There was no traffic spike. A 'performance' change last sprint cranked the socket send buffers way up (SO_SNDBUF) and increased an internal write-queue size to 'smooth out' uploads. NIC utilization is moderate; CPU and memory are fine. How do you triage and mitigate?

It covers

  • Saturation
  • Bufferbloat
  • Tcp queueing