Cold cache stampede after restart

On-call incidentHard30 min

Your task

Your read path is fronted by a large Redis cluster that normally absorbs 98% of traffic. After a planned cluster-wide Redis restart for a version upgrade at 02:00, the cache comes back completely empty. The instant the app fleet starts serving again, origin DB CPU pins at 100%, p99 jumps 40x, and the DB shows thousands of identical queries per popular key, the warm-up never converges and the DB stays saturated for 20+ minutes. CPU on the app tier is fine. How do you triage and mitigate the cold-cache stampede?

It covers

  • Cache fill
  • Cluster restart
  • Single flight