Synchronous logging blocking request threads

On-call incidentMedium30 min

Your task

After a debugging session left log level at DEBUG in prod, your service's p99 jumped from 20ms to 300ms under load while p50 barely moved. Flame graphs show request threads spending significant time inside synchronous logging calls, and disk write latency on the log volume is elevated and bursty. CPU is moderate. The logging is configured synchronous and the appender fsyncs. Throughput is down. How do you triage and mitigate?

It covers

  • Log flush
  • Synchronous io
  • Blocking logging