On-callHardoc-g347

Subject Data incidentsLevel Senior–Staff~35 minCommon in Reliability & on-call interviewsIndustries Technology

Question

A session/token store (Cassandra with per-row TTLs) starts logging users out early and dropping short-lived grant tokens before they should expire, beginning ~03:30. Dashboards: error rate flat, but 'session expired' events jumped and a few cache layers show entries vanishing ahead of their TTL. Context: one of three new Cassandra nodes added last night came up with NTP not yet synced and its clock is ~7 minutes *ahead*. How do you triage, stop the premature data expiry, and recover affected users?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Learn the concepts

Diagram & narrate the incident

Loading whiteboard…

Run or narrate your approach, then ask the coach.