Canary blind to currency
Your task
Your automated canary for v88 of the payments service runs at 10:00: 1 canary pod + 1 baseline pod, 5% traffic each. The canary analysis goes green and auto-promotes to 100% at 10:20. By 10:35 you're paged: 2% of payment captures are silently failing with a downstream gateway returning HTTP 402, but only for one currency (JPY). The canary dashboards during the 20-minute window showed identical error rates between canary and baseline. The diff in v88 changed how amounts are formatted before sending to the gateway. Triage and mitigate.
It covers
- Canary analysis
- Metrics
- Deploy