Sessions expire ahead of schedule

On-call incidentHard35 min

Your task

A session/token store (Cassandra with per-row TTLs) starts logging users out early and dropping short-lived grant tokens before they should expire, beginning ~03:30. Dashboards: error rate flat, but 'session expired' events jumped and a few cache layers show entries vanishing ahead of their TTL. Context: one of three new Cassandra nodes added last night came up with NTP not yet synced and its clock is ~7 minutes ahead. How do you triage, stop the premature data expiry, and recover affected users?

It covers

  • Clock skew
  • Ntp
  • Ttl expiry
  • Premature deletion