Latency cliffs every 90 seconds

On-call incidentHard40 min

Your task

A JVM service (G1GC, 32GB heap) develops periodic latency cliffs: every ~90 seconds, p99 jumps to 4-8s for a second or two, then recovers. GC logs show occasional 'Full GC (Allocation Failure)' events with pauses of 5+ seconds, alongside normal short young-gen pauses. Old-gen occupancy climbs steadily between Full GCs (sawtooth that doesn't fully reset). A deploy two days ago added an in-memory LRU-ish cache that holds rendered response fragments. Heap-used trend is slowly creeping up day over day. Triage and mitigate.

It covers

  • Jvm
  • Stop the world
  • Allocation rate