Elasticsearch OOM from expensive aggregation

On-call incidentHard40 min

Your task

At 17:05 your Elasticsearch-backed search service starts crash-looping: data nodes hit OOM and restart, the cluster goes yellow then red, and as each node restarts it dies again within a minute. APM shows that just before each crash there's a single very expensive query, a deeply nested aggregation with a huge terms cardinality and a wide date range, submitted repeatedly by one client. Because it's persisted to a saved-search dashboard, it keeps getting re-issued. How do you triage and break the loop?

It covers

  • Resource exhaustion
  • Oom
  • Crash loop