Recommendations show sold-out items

On-call incidentMedium30 minAmazon style

Your task

Your storefront caches each user's top-N recommendations in a key-value cache to avoid re-scoring on every page load. Support tickets spike: users complain the homepage keeps recommending items that are out of stock or that they already bought hours ago, and merchandising says a flash-sale collection that ENDED is still being recommended. The model and scoring path are healthy and fast. Dashboards: the recommendation cache has a 24-hour TTL and a 99% hit rate; there is no event-driven invalidation when inventory, purchases, or campaign state change; the flash sale ended at 00:00 but cached recs from before then are still being served. How do you triage and respond?

It covers

  • Recommendation cache
  • Stale items
  • Ttl
  • Invalidation