Checkout errors return minutes after every rollback

On-call incidentEasy15 min

Your task

It is 14:46. The checkout service runs on Kubernetes. Flux, a tool that applies the manifests in a Git repo to the cluster, deploys it from the deploy repo. Version 3.18 went out at 14:02. This alert has fired 3 times since then, most recently at 14:45:

CheckoutErrorRate: 5xx above 2% for 2 minutes (current 11%)

What you see:

  • 3.18 pods log KeyError: 'tax_region' on about 11% of checkouts. 3.17 never logged it.
  • You ran kubectl rollout undo deployment/checkout at 14:14 and at 14:33. Both times the errors fell below 0.5% within 2 minutes. They came back at 14:23 and at 14:43.
  • The Deployment's image reads checkout:3.18 again. In the team channel, nobody has released or touched checkout since 14:02.
  • The newest commit in deploy is from 14:01: checkout: bump image to 3.18.

How do you get 3.17 to stay in production, and what has to happen before 3.18 goes out again?

It covers

  • Flux
  • Kubernetes deployments
  • Release process

About this question

Checkout errors return minutes after every rollback

FluxKubernetes deploymentsRelease process

Model answer

Free on every account

First move: change what the deploy repo says, so 3.17 stays. Revert the 14:01 commit, push it, and have Flux apply it now:

git revert <14:01 commit>
git push
flux reconcile kustomization checkout --with-source

Run flux get kustomizations if yours has another name. Flux rolls checkout to 3.17, and errors should fall below 0.5% within 2 minutes, as they did after each undo. Then watch past Flux's next run to see it hold.

If a review or a protected branch will take more than a few minutes, stop the harm first. Run flux suspend kustomization checkout, then kubectl rollout undo deployment/checkout once more. Suspending pauses everything that Kustomization manages, not only checkout, so other changes in it stop applying too. Get the revert merged, run flux reconcile source git <name> so Flux has fetched it, then run flux resume kustomization checkout. Resume before Flux has fetched it, and 3.18 comes back.

On either route, if Flux's image automation wrote the 14:01 commit, run flux suspend image update <name> too, or it can bump the tag again and undo your revert.

This does not fix the bug in 3.18, and it does not bring back the checkouts that failed. Pull those from the logs for support.

Read the signals. Both undos worked: errors fell within 2 minutes, so 3.17 is good and the fault is in 3.18. Both then stopped working with nobody on the team doing anything. The Deployment went back to 3.18, and the repo's newest commit still says 3.18. That is how Flux works. At each interval it applies what the repo says and corrects anything in the cluster that differs. kubectl rollout undo changes only the cluster, so the next run overwrites it.

The times fit a 10-minute interval counted from the 14:02 release, with runs near 14:12, 14:22, 14:32 and 14:42. The undo at 14:14 lasted until the 14:22 run, and errors were back by 14:23. The undo at 14:33 lasted until the 14:42 run, and errors were back by 14:43. The runs at 14:12 and 14:32 found 3.18 already live and changed nothing. The alert's 2-minute wait explains its 3 firings: about 14:04, 14:25 and 14:45, each 2 minutes after errors began or came back.

Other causes do not fit. A teammate redeploying would show in the channel. An autoscaler changes the pod count, not the image. A second pipeline that applies manifests could do this too, and the Confirm step rules it out.

YesNoYesNoErrors return after eachundoImage back on 3.18 withno release?Flux applies the repo,and the repo says 3.18Look for another causeof the errorsCan a revert merge inminutes?Revert in Git andreconcile nowSuspend Flux and undo,then revert, fetch andresumeWatch past the next FluxrunFix the bug, ship a newversion through the repo

Confirm. kubectl get kustomization checkout -n flux-system -o jsonpath='{.spec.interval}' shows the interval, which should be 10m or 10m0s. kubectl get deployment checkout -o yaml --show-managed-fields lists manager: kustomize-controller against the image field, unless your own undo wrote it last. flux events --for Kustomization/checkout shows its recent runs. Identical events merge, so Deployment/<namespace>/checkout configured shows once, with a count and a last seen time near 14:42, the run that last changed the image back. After the revert, the image still reads checkout:3.17 past the next run.

Fix. Before that code goes out again:

  • Find which checkouts have no tax_region. About 11% do, so this is a common path, not a rare one. Handle it, and add a test built from a real payload without the key.
  • Ship the fix as a new version, such as 3.19, through the repo. Do not revert the revert, which brings back the same broken image.
  • Send it to a small share of traffic first if you can, and watch the KeyError count.

Follow-ups.

  • Write it down: under Flux, a rollback is a revert in deploy, and flux suspend is the emergency route when review is slow.
  • Make sure on-call can get a revert merged out of hours.
  • Send Flux's notifications to the team channel, so an image change nobody made by hand is easy to see.

Concept guide

Related questions