Metric label cardinality explosion

On-call incidentHard35 min

Your task

A service's RSS grows ~80MB/hour and it gets OOMKilled every ~30 hours; restart resets it. Heap profiles show the memory is dominated by one in-process map keyed by a metric label derived from a request attribute, and the number of distinct keys grows without bound. The growth rate jumped sharply two days ago when a client started sending a unique request ID into a header that the service (incorrectly) uses as a metric label dimension, so every request creates a new label series that's retained forever. Request rate is flat. Triage and mitigate.

It covers

  • Cardinality
  • Unbounded keys
  • Metrics labels