Checkout errors return minutes after every rollback
Your task
It is 14:46. The checkout service runs on Kubernetes. Flux, a tool that applies the manifests in a Git repo to the cluster, deploys it from the deploy repo. Version 3.18 went out at 14:02. This alert has fired 3 times since then, most recently at 14:45:
CheckoutErrorRate: 5xx above 2% for 2 minutes (current 11%)
What you see:
- 3.18 pods log
KeyError: 'tax_region'on about 11% of checkouts. 3.17 never logged it. - You ran
kubectl rollout undo deployment/checkoutat 14:14 and at 14:33. Both times the errors fell below 0.5% within 2 minutes. They came back at 14:23 and at 14:43. - The Deployment's image reads
checkout:3.18again. In the team channel, nobody has released or touched checkout since 14:02. - The newest commit in
deployis from 14:01:checkout: bump image to 3.18.
How do you get 3.17 to stay in production, and what has to happen before 3.18 goes out again?
It covers
- Flux
- Kubernetes deployments
- Release process
About this question
Checkout errors return minutes after every rollback
Model answer
Free on every accountFirst move: change what the deploy repo says, so 3.17 stays. Revert the 14:01 commit, push it, and have Flux apply it now:
git revert <14:01 commit> git push flux reconcile kustomization checkout --with-source
Run flux get kustomizations if yours has another name. Flux rolls checkout to 3.17, and errors should fall below 0.5% within 2 minutes, as they did after each undo. Then watch past Flux's next run to see it hold.
If a review or a protected branch will take more than a few minutes, stop the harm first. Run flux suspend kustomization checkout, then kubectl rollout undo deployment/checkout once more. Suspending pauses everything that Kustomization manages, not only checkout, so other changes in it stop applying too. Get the revert merged, run flux reconcile source git <name> so Flux has fetched it, then run flux resume kustomization checkout. Resume before Flux has fetched it, and 3.18 comes back.
On either route, if Flux's image automation wrote the 14:01 commit, run flux suspend image update <name> too, or it can bump the tag again and undo your revert.
This does not fix the bug in 3.18, and it does not bring back the checkouts that failed. Pull those from the logs for support.
Read the signals. Both undos worked: errors fell within 2 minutes, so 3.17 is good and the fault is in 3.18. Both then stopped working with nobody on the team doing anything. The Deployment went back to 3.18, and the repo's newest commit still says 3.18. That is how Flux works. At each interval it applies what the repo says and corrects anything in the cluster that differs. kubectl rollout undo changes only the cluster, so the next run overwrites it.
The times fit a 10-minute interval counted from the 14:02 release, with runs near 14:12, 14:22, 14:32 and 14:42. The undo at 14:14 lasted until the 14:22 run, and errors were back by 14:23. The undo at 14:33 lasted until the 14:42 run, and errors were back by 14:43. The runs at 14:12 and 14:32 found 3.18 already live and changed nothing. The alert's 2-minute wait explains its 3 firings: about 14:04, 14:25 and 14:45, each 2 minutes after errors began or came back.
Other causes do not fit. A teammate redeploying would show in the channel. An autoscaler changes the pod count, not the image. A second pipeline that applies manifests could do this too, and the Confirm step rules it out.
Confirm. kubectl get kustomization checkout -n flux-system -o jsonpath='{.spec.interval}' shows the interval, which should be 10m or 10m0s. kubectl get deployment checkout -o yaml --show-managed-fields lists manager: kustomize-controller against the image field, unless your own undo wrote it last. flux events --for Kustomization/checkout shows its recent runs. Identical events merge, so Deployment/<namespace>/checkout configured shows once, with a count and a last seen time near 14:42, the run that last changed the image back. After the revert, the image still reads checkout:3.17 past the next run.
Fix. Before that code goes out again:
- Find which checkouts have no
tax_region. About 11% do, so this is a common path, not a rare one. Handle it, and add a test built from a real payload without the key. - Ship the fix as a new version, such as 3.19, through the repo. Do not revert the revert, which brings back the same broken image.
- Send it to a small share of traffic first if you can, and watch the
KeyErrorcount.
Follow-ups.
- Write it down: under Flux, a rollback is a revert in
deploy, andflux suspendis the emergency route when review is slow. - Make sure on-call can get a revert merged out of hours.
- Send Flux's notifications to the team channel, so an image change nobody made by hand is easy to see.