Code RoomRace condition in refactored order write path
HardPrep Room Coding #2892

Race condition in refactored order write path

On-callDatabases & SQLReliability & on-callSenior–Staff~45 min

At 07:30 your e-commerce order service starts showing impossible data: some orders have `total_amount` that doesn't match the sum of their line items, and a few hundred orders reference `customer_id`s that don't exist. The app is up. Digging in: a deploy at 06:55 shipped a refactor of the order-write path that, under a specific concurrent-checkout race, wrote orders with partially-applied updates (it split one transaction into two non-atomic writes). New bad rows are STILL being created as traffic flows. Your DB has PITR available, and a read replica is in sync. The bad and good rows are interleaved in time (legit orders are being placed continuously alongside the corrupt ones). Walk through containment, recovery of the corrupted rows, and prevention — note what makes this harder than a single bad batch UPDATE.

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

0:00 of about 45 min
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.