Cache hit rate crashes
Your task
It's 11:02 UTC. PagerDuty fires: product-catalog API p99 latency jumped from 40ms to 4.8s and the origin Postgres read replica is at 100% CPU with a queue of 900 active queries, almost all the identical SELECT ... FROM catalog WHERE id IN (...) for the homepage's top-50 SKUs. The Redis dashboard shows a hit rate that cratered from 98% to 11% about 90 seconds ago, and the keyspace shows the catalog:home:* keys are missing. There was no deploy. Traffic is normal for a Tuesday. A teammate mentions that an hour ago someone shortened the homepage cache TTL from 1h to 5m "to make merchandising changes show up faster." The replica is now so saturated that even uncached requests are timing out, and the error rate is climbing. Walk me through how you triage and stabilize this, then make it durable.
It covers
- Cache stampede
- Thundering herd