Code Room
On-callHard
Question
Your product pages are served from a read-through cache (Redis, per-product key, 1h TTL); on a DB write you publish an invalidation that deletes the affected key. At 13:00 a backfill started normalizing a field on all 50M products, updating ~5k rows/sec. Within minutes the database CPU pegs at 100%, page p99 spikes to seconds, and the cache hit rate craters — even for products the backfill hasn't touched. Dashboards: Redis healthy, DB connections maxed, query rate to the DB is ~50x baseline. How do you triage and stop the meltdown?
What a strong answer looks like
Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.
Learn the concepts
Loading whiteboard…
Run or narrate your approach, then ask the coach.