Secret rotation breaks webhook verification

On-call incidentHard40 minStripe style

Your task

Responding to a suspected leak, at 14:00 you rotate a shared HMAC webhook-signing secret in the vault and revoke the old value. Within 90 seconds, webhook verification across 30+ consumer services starts failing, signatures computed with the new secret are rejected by consumers still holding the old one cached in memory (they only re-read the secret on restart, which can be hours). Inbound partner webhooks are now being dropped. You need the leak contained AND traffic restored. Walk through how you triage, restore service, and run a rotation that doesn't cause this, without leaving the leaked secret valid longer than necessary.

It covers

  • Rotation rollout
  • Secret caching
  • Graceful rotation
  • Dual secret