Cache invalidation storm from backfill

On-call incidentHard35 min

Your task

Your product pages are served from a read-through cache (Redis, per-product key, 1h TTL); on a DB write you publish an invalidation that deletes the affected key. At 13:00 a backfill started normalizing a field on all 50M products, updating ~5k rows/sec. Within minutes the database CPU pegs at 100%, page p99 spikes to seconds, and the cache hit rate craters, even for products the backfill hasn't touched. Dashboards: Redis healthy, DB connections maxed, query rate to the DB is ~50x baseline. How do you triage and stop the meltdown?

It covers

  • Cache stampede
  • Invalidation storm
  • Thundering herd
  • Backfill