Queue depth increases tail latency

On-call incidentHard35 minGoogle style

Your task

During traffic surges your service's p99 balloons to 4-6s while p50 stays low, and notably the very old requests are the ones timing out, by the time the server starts processing a request, the client has already given up. You recently raised the inbound accept/request queue depth and the worker queue size 'so we stop dropping requests under load.' CPU is at 100% during the surge but throughput (completed RPS) actually fell vs. before the change. How do you triage and fix?

It covers

  • Buffer bloat
  • Queueing
  • Load shedding