Elasticsearch OOM from expensive aggregation
Your task
At 17:05 your Elasticsearch-backed search service starts crash-looping: data nodes hit OOM and restart, the cluster goes yellow then red, and as each node restarts it dies again within a minute. APM shows that just before each crash there's a single very expensive query, a deeply nested aggregation with a huge terms cardinality and a wide date range, submitted repeatedly by one client. Because it's persisted to a saved-search dashboard, it keeps getting re-issued. How do you triage and break the loop?
It covers
- Resource exhaustion
- Oom
- Crash loop