A one-line discount fix crashes every new pod

On-call incidentEasy15 min

Your task

It is 16:10. Discount codes have been failing for some customers since this morning. You wrote a one-line fix in pricing/discounts.py, and CI deployed it at 15:55. The pricing service runs 6 pods on Kubernetes, and this is its first new image in 2 years. This alert has been firing since 16:05:

RolloutStalled: pricing rollout made no progress for 10 minutes (threshold 10 minutes)

What you see:

  • kubectl get pods -l app=pricing shows 5 old pods Running and 3 new pods in CrashLoopBackOff, which means a container keeps crashing and Kubernetes waits longer before each restart.
  • Each new pod's log has a traceback that ends with File "/app/pricing/receipts.py", line 3, in <module> and ModuleNotFoundError: No module named 'cgi'. The earlier lines of that traceback include paths under /usr/local/lib/python3.14/site-packages/gunicorn/.
  • receipts.py has not changed in 2 years. requirements.txt pins every package to an exact version, and it has not changed either. CI ran the unit tests on the CI runner's Python 3.12, and they passed.
  • The Dockerfile starts with FROM python:3-slim. On an old pod, python --version prints Python 3.12.4.

How do you get the discount fix to customers today, and what do you change so the next image build does not do this?

It covers

  • Docker builds
  • Python
  • Kubernetes rollouts

About this question

A one-line discount fix crashes every new pod

Docker buildsPythonKubernetes rollouts

Model answer

Free on every account

First move: rebuild the same one-line fix on the Python the old pods run. The rollout stopped by itself, and 5 old pods still serve customers. A pod in CrashLoopBackOff is not ready while it waits to restart, so it gets no traffic then. With no readiness probe it can catch a few requests during each brief restart, so check the error rate too. The live harm is the discount bug, plus one pod less capacity, so check CPU and latency on the 5. Change the first line of the Dockerfile:

FROM python:3.12.4-slim

Commit it with the fix and let CI deploy. The new image replaces the crashing pods, so you do not need to undo first. If the build will take a while, kubectl rollout undo deployment/pricing brings back the sixth old pod and clears the crash loop, and the next deploy rolls forward from there. An undo alone does not help customers, because the old image still has the discount bug.

Watch kubectl rollout status deployment/pricing, then try a discount code that failed this morning.

python:3.12-slim also works and takes newer 3.12 patches, but today it also moves the OS from Debian 12 to Debian 13. python:3.12-slim-bookworm keeps the old OS. If the old image is still in the registry, building FROM that image and copying in discounts.py is the smallest change of all.

What this does not fix: receipts.py still imports a module that Python 3.13 and later do not have. python:3.12.4-slim was last rebuilt in August 2024, so it has had no Python or OS patches since, no better than the old pods. The 3.12 line gets security fixes until about October 2028, but you take them only by moving the pin. That upgrade comes later, as its own change. Removing the import, or adding the standard-cgi package from PyPI, might also start the pods. But it puts a hotfix on a Python this service has never run on, so keep it for the case where no 3.12 image can be built.

Read the signals. The traceback never touches discounts.py. It stops at receipts.py line 3, which has not changed in 2 years, and the pinned packages did not change either. What changed is Python. The traceback's paths say python3.14, and an old pod prints 3.12.4. FROM python:3-slim means the newest Python 3 at build time. That tag moves with each release, so the first build in 2 years jumped from 3.12 to 3.14. Since mid-October 2026 a build gets 3.15. The cgi module left the standard library in Python 3.13.

CI could not see this. Its unit tests ran on the runner's Python 3.12, not inside the image.

The pod counts show why customers are still served. With 6 replicas and the default 25% settings, a rollout may run 8 pods and must keep 5 available. It stopped 1 old pod and started 3 new ones, and it has gone no further while those crash.

Other causes do not fit. A bad fix would fail in discounts.py or its tests. A broken package would show in that package's path. A missing system library would name that library, not a standard Python module.

YesNoNew pods crash, 5 oldpods serveDoes the traceback touchthe changed file?Look at the fix itselfCompare Python in oldpods and new image3.12.4 against 3.14, cgiis goneRebuild the fix on theold PythonTest a code that failedthis morningExact base image, testsinside the image

Confirm. Run the new image outside the cluster. docker run --rm <new image> python --version prints 3.14, and docker run --rm <new image> python -c "import cgi" fails with the same error. The same import on python:3.12.4-slim works. kubectl describe deployment pricing shows ProgressDeadlineExceeded under Progressing, which matches the alert: the default deadline is 600 seconds, and 15:55 plus 10 minutes is 16:05.

Fix.

  • Name an exact Python in the Dockerfile, ideally with a digest, such as python:3.12.4-slim@sha256:..., so a rebuild gives the same base. Let a tool such as Renovate open a pull request when a newer one appears.
  • Run the tests inside the built image in CI, and start the container and call its health check before any deploy.
  • Rebuild on a schedule, say monthly, so changes arrive in small steps and not as a 2-year jump.

Follow-ups.

  • Open a ticket with an owner to replace cgi and move to a current Python. The Python 3.13 release notes list replacements.
  • Search your other Dockerfiles for tags like :3-slim and :latest.

Concept guide

Related questions