Stale personalized content after rollback

On-call incidentMedium35 min

Your task

v90 of a content API deployed at 10:00 and briefly served responses with a wrong Cache-Control (it set 'public, max-age=86400' on personalized, user-specific pages). You catch it and roll back to v89 at 10:08. But users keep reporting they see other people's personalized content for hours after the rollback, even though the code is correct now and the deploy dashboard is green. Triage and mitigate.

It covers

  • Cache poisoning
  • Rollback ineffective
  • Cdn