GC pauses after serialization refactor

On-call incidentMedium30 min

Your task

A Go JSON API service shows latency jitter that started after a deploy: p50 is fine but p99 is sawtoothing and the service occasionally pauses. runtime metrics show GC running very frequently (many cycles per second) and gc_pause_ns is elevated; allocation rate (go_memstats_alloc_bytes rate) roughly tripled after the deploy. CPU usage also rose ~25% but the work done is the same. The deploy 'refactored a hot serialization path.' Heap size is stable, no leak. Triage and mitigate.

It covers

  • Go runtime
  • Allocation spike
  • Gogc