Cache misses from nonexistent products

On-call incidentHard35 min

Your task

Starting at 22:45, your API's cache-hit ratio is normal at 88% but the origin database load has quietly doubled and a specific set of product-lookup endpoints is slow. Logs show a flood of requests for product IDs that don't exist (random-looking integers), each one a cache miss that falls through to a DB query returning no rows. The traffic is coming from a wide spread of IPs. No deploy happened. How do you triage and mitigate?

It covers

  • Stampede
  • Cache penetration
  • Null caching