In-flight requests lost

On-call incidentHard40 minStripe style

Your task

You flip a payment-processing API from blue to green at 14:00 by switching the LB target group, then immediately scale blue to zero. Green is healthy. But over the next hour, reconciliation flags ~200 payments stuck in 'PROCESSING' that never reached a terminal state. These all correspond to requests that were in flight on blue at 14:00. Blue handled an async finalize step a few seconds after the initial response. Triage, explain the failure, and describe how to recover and prevent it.

It covers

  • Stateful
  • In flight
  • Draining