Clock skew token rejection

On-call incidentMedium30 min

Your task

Starting 09:30, ~4% of logins fail with 'token used before issued (iat)' or 'token expired'. Dashboards: failures are concentrated on one Kubernetes node pool; the affected pods all run on hosts that came up after an autoscaler scale-out at 09:25; chrony/NTP metric shows those hosts ~45s ahead of the rest of the fleet; auth success rate elsewhere is normal. A new VM image was rolled into the autoscaling group yesterday. How do you triage and mitigate?

It covers

  • Ntp
  • Token expiry
  • Tls