On-callHardoc-g015

Subject Cache incidentsLevel Senior–Staff~35 minCommon in Reliability & on-call interviewsIndustries Technology, Software development

Question

At 11:20 your primary Redis (ElastiCache) cluster failed over to a replica after a node fault. The failover took 25 seconds. During and after it, your search service didn't just blip — it stayed degraded for 8 minutes: connection errors to Redis for 25s, then a sustained period where origin DB CPU was pinned and p99 was 10x normal, slowly recovering. Your Redis client has a 50ms connect timeout but retries with no backoff. How do you triage and what are the failure modes to fix?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Learn the concepts

Diagram & narrate the incident

Loading whiteboard…

Run or narrate your approach, then ask the coach.