Garbage collection pauses degrade

On-call incidentHard28 min

Your task

Your JVM search service degrades under a gradual traffic increase: p99 climbs and a few instances become briefly unresponsive, then recover, in a rolling pattern. CPU shows frequent spikes to 100% but the request rate that triggers it is only ~20% above last month. GC logs show full GCs every few seconds with multi-hundred-millisecond pauses, heap usage sawtoothing near the max, and old-gen filling faster than it's collected. A recent change added a large per-request in-memory result cache. How do you triage and mitigate?

It covers

  • Memory pressure
  • Gc thrash
  • Saturation