Cache stampede after rolling restart

On-call incidentMedium30 min

Your task

Right after a 16:00 deploy, your product-detail service falls over: p99 latency goes from 90ms to 6s, the database CPU pins at 100%, and the error rate spikes to 20% for about three minutes before everything settles back to normal on its own. The deploy was a routine code change, but it required a rolling restart of the Memcached fleet to pick up new client config. There were no schema or query changes. How do you triage, mitigate the immediate pain, and prevent the next deploy from doing this?

It covers

  • Cold cache
  • Deploy
  • Warmup