Code RoomCache invalidation storm from backfill
HardPrep Room Coding #2691

Cache invalidation storm from backfill

On-callReliability & on-callSenior–Staff~35 min

Your product pages are served from a read-through cache (Redis, per-product key, 1h TTL); on a DB write you publish an invalidation that deletes the affected key. At 13:00 a backfill started normalizing a field on all 50M products, updating ~5k rows/sec. Within minutes the database CPU pegs at 100%, page p99 spikes to seconds, and the cache hit rate craters — even for products the backfill hasn't touched. Dashboards: Redis healthy, DB connections maxed, query rate to the DB is ~50x baseline. How do you triage and stop the meltdown?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

0:00 of about 35 min
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.