Metric label cardinality explosion
Your task
A service's RSS grows ~80MB/hour and it gets OOMKilled every ~30 hours; restart resets it. Heap profiles show the memory is dominated by one in-process map keyed by a metric label derived from a request attribute, and the number of distinct keys grows without bound. The growth rate jumped sharply two days ago when a client started sending a unique request ID into a header that the service (incorrectly) uses as a metric label dimension, so every request creates a new label series that's retained forever. Request rate is flat. Triage and mitigate.
It covers
- Cardinality
- Unbounded keys
- Metrics labels