Cache invalidation storm from backfill
Your task
Your product pages are served from a read-through cache (Redis, per-product key, 1h TTL); on a DB write you publish an invalidation that deletes the affected key. At 13:00 a backfill started normalizing a field on all 50M products, updating ~5k rows/sec. Within minutes the database CPU pegs at 100%, page p99 spikes to seconds, and the cache hit rate craters, even for products the backfill hasn't touched. Dashboards: Redis healthy, DB connections maxed, query rate to the DB is ~50x baseline. How do you triage and stop the meltdown?
It covers
- Cache stampede
- Invalidation storm
- Thundering herd
- Backfill