Cache hit ratio collapses

On-call incidentHard35 min

Your task

Over the last 30 minutes your Redis cache (maxmemory set, allkeys-lru eviction) has degraded: hit ratio fell from 95% to 60%, evicted_keys is climbing by tens of thousands per second, the origin DB load is doubling, and Redis p99 GET latency rose from 0.4ms to 9ms. used_memory is pinned at maxmemory. A new feature shipped this morning that caches per-user personalized search results keyed by user_id + a serialized query object, with no explicit TTL. How do you triage and mitigate?

It covers

  • Eviction storm
  • Memory pressure