On-callHardoc-g014

Subject Cache incidentsLevel Senior–Staff~35 minCommon in Reliability & on-call · Distributed systems interviewsIndustries Technology, Software development

Question

Starting at 22:45, your API's cache-hit ratio is normal at 88% but the origin database load has quietly doubled and a specific set of product-lookup endpoints is slow. Logs show a flood of requests for product IDs that don't exist (random-looking integers), each one a cache miss that falls through to a DB query returning no rows. The traffic is coming from a wide spread of IPs. No deploy happened. How do you triage and mitigate?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Learn the concepts

Diagram & narrate the incident

Loading whiteboard…

Run or narrate your approach, then ask the coach.