Canary blind to currency

On-call incidentMedium30 minStripe style

Your task

Your automated canary for v88 of the payments service runs at 10:00: 1 canary pod + 1 baseline pod, 5% traffic each. The canary analysis goes green and auto-promotes to 100% at 10:20. By 10:35 you're paged: 2% of payment captures are silently failing with a downstream gateway returning HTTP 402, but only for one currency (JPY). The canary dashboards during the 20-minute window showed identical error rates between canary and baseline. The diff in v88 changed how amounts are formatted before sending to the gateway. Triage and mitigate.

It covers

  • Canary analysis
  • Metrics
  • Deploy