Cache stampede on viral content

On-call incidentHard40 min

Your task

A celebrity tweets a link to your article page at 20:30. Within seconds, origin DB CPU saturates, p99 on the article endpoint goes from 50ms to 9s, and your Redis dashboard shows hit-rate collapsing from 98% to 40% with a burst of identical key MISSes. The CDN dashboard shows a sudden flood of requests for the exact same URL. Nothing was deployed today. The cache TTL for article HTML is 60 seconds. How do you triage this and stop the origin from melting?

It covers

  • Cache stampede
  • Redis
  • Origin