Bad Data Deploy

When code rollback doesn't save you because the database is already corrupted.

The idea

A bad code deployment can usually be fixed in seconds by hitting "Rollback". However, if the bad code wrote corrupted or backwards-incompatible data to the database, rolling back the code will cause the old code to crash when it reads the new, corrupted data. You are now stuck: you cannot roll forward, and you cannot roll backward.

How it works (Schema Evolution)

To avoid this, all database schema changes or data migrations must be done in backward-compatible steps (the Expand/Contract pattern). Never introduce a data format that older code cannot safely ignore or parse.

# Scenario: Migrating "name" (String) to "first_name" / "last_name"

# WRONG WAY (Breaks Rollback):
# V2 Code deploys, deletes "name" column, adds "first_name".
# If V2 crashes, rolling back to V1 fails because "name" is missing!

# RIGHT WAY (Expand / Contract):
# Step 1 (Expand): Deploy V1.5. Adds "first_name" column, but keeps "name". 
#                  Code writes to BOTH, reads from "name".
# Step 2 (Migrate): Backfill old rows so they have "first_name".
# Step 3 (Cutover): Deploy V2. Reads/writes ONLY to "first_name".
# Step 4 (Contract): Once V2 is stable for weeks, drop the "name" column.

Cost

Writing backward-compatible data deployments significantly slows down feature velocity. Instead of a quick database tweak, a simple column rename might require 4 separate deployments spanning several days.

Watch out for

• Enum additions: Adding a new enum value (e.g. `status = 'PENDING_REVIEW'`) is dangerous. If V2 writes it, and you rollback to V1, V1 will crash when reading an enum value it doesn't recognize. V1 must be taught to ignore unknown enums before V2 deploys.
• Cache Poisoning: Bad data doesn't just live in the DB. If bad code writes a corrupted JSON object to Redis, you must flush the cache as part of your incident response.