Cache collapse metastable failure
Your task
At 20:00 your social feed service falls over and STAYS down even after the traffic spike that started it has passed. Dashboards: DB CPU pinned at 100%; cache hit rate collapsed from 95% to 20% at 19:58; request rate to the DB is 10x normal; restarting app pods brings them up healthy for ~30s then they fall over again. Earlier at 19:57 there was a brief 3x traffic spike (a celebrity post) that has since subsided to normal. How do you triage and mitigate?
It covers
- Cache stampede
- Load amplification
- Load shedding