Customers pay for annual plans but stay on Free

On-call incidentEasy15 min

Your task

It is 00:25 UTC on 29 February. Your app is written in Python 3.12 and takes payments through Stripe. When a customer buys a plan, Stripe sends a webhook, and the billing service creates the subscription and sets when it ends. Stripe retries a failed webhook for up to 3 days. This alert has been firing since 00:11:

BillingWebhookErrors: 5xx on /webhooks/stripe above 5% for 10 minutes (current 38%)

What you see:

  • Every failed webhook logs ValueError: day is out of range for month at billing/plans.py:88 in annual_end_date. Monthly plan webhooks all succeed.
  • The errors began at 00:00:04 UTC. Traffic is normal, and the last release was 5 days ago.
  • Stripe shows 104 annual plan payments since midnight. Your database has no new annual subscription since 23:59 yesterday.
  • Support has 12 tickets saying "I paid, but my account still says Free".

How do you get these customers the plan they paid for tonight, and what do you check before you go back to sleep?

It covers

  • Stripe webhooks
  • Billing
  • Subscriptions

About this question

Customers pay for annual plans but stay on Free

Stripe webhooksBillingSubscriptions

Model answer

Free on every account

First move: ship a fix to annual_end_date, then resend the failed events. A rollback probably does nothing. Running fine for 5 days proves nothing here, because this line only breaks on this date, so check git blame on billing/plans.py:88. If neither the line nor its caller changed in the last release, a rollback cannot help. If that release changed either, a rollback can stop new failures, and customers who already paid still need the steps below. Read billing/plans.py:88. It almost certainly replaces the year in today's date, and next year has no 29 February. Decide the rule and fall back to it:

def annual_end_date(start):
    try:
        return start.replace(year=start.year + 1)
    except ValueError:
        return start.replace(year=start.year + 1, day=28)

dateutil's relativedelta(years=1) also gives 28 February, if the package is already in your app.

If Stripe bills the renewal, read its period end on one of today's subscriptions and match it, so nobody loses access before then. Stripe's docs say that if a month lacks the billing day, it bills on the month's last day, so expect 28 February, which this code gives. 1 March also works as a product rule, 1 day past the charge. Pass in the time of the payment, not the current time, so a retry tomorrow gets the same date. Add a test for 29 February and ship it on your fastest path. New buyers keep failing at about 4 a minute until then, and recover the same way.

What this does not fix: customers who have already paid are still on Free. Stripe's retries back off, so the next one can be hours away. After the deploy, resend the failed events: list them with the Events API using delivery_success=false and the event types you handle, paging through all of them since it returns 10 by default. Then use Resend in the Stripe Dashboard or stripe events resend <event_id> --webhook-endpoint=<endpoint_id> --live. The CLI runs in a sandbox unless you pass --live, and --confirm skips its prompt in a loop.

First check the handler is safe to run twice, because a resend does not cancel Stripe's automatic retries. It must skip an event it has already handled and return 200, which also stops Stripe's retries for that event. Key this on the ID of the object in the event, such as the checkout session, since Stripe can occasionally send 2 separate events for one object. A check before the insert is not enough, because a retry and a resend can arrive together. Put a unique constraint on that ID, in a new table so nothing busy locks, and insert it in the same transaction as the subscription. The second transaction then fails on the constraint, and the handler returns 200. If it has no such guard, add one in this same fix. Check too that the failed runs left no half-made rows behind.

Read the signals. The error names a date problem inside a function that works out an end date. It began 4 seconds into 29 February, with normal traffic, so the date set it off, not a deploy. The last release was 5 days ago, and git blame says whether it changed this line. Monthly plans succeed, since 29 March exists. Stripe shows 104 payments and your database shows no new annual subscriptions, so the money was taken and the plan was never made.

Stripe retries for up to 3 days, so no event is lost yet.

YesNo, it is olderAnnual webhooks failwith a date errorDid the last releasechange this code?Roll back that releaseFix the date code, makethe handler safeShip it, then resend thefailed eventsCompare Stripe'spayments with newsubscriptionsSearch other date code,answer the tickets

Confirm. In a Python 3.12 shell, from datetime import date; date(2028, 2, 29).replace(year=2029) raises the same ValueError. After the deploy, the 5xx rate on /webhooks/stripe falls back to normal.

Before you go back to sleep, check:

  • Annual payments in Stripe since midnight equal new annual subscriptions in your database, with no customer holding 2.
  • The webhook's Event deliveries tab shows new attempts succeeding. A resent event can still show a retry pending until it arrives and your handler returns 200, so the count above is the real check.
  • The new rows have the end date you chose.
  • Nothing else does the same arithmetic. Search for replace(year= and similar in renewals, trials, invoices and reports, and in any job that runs later today.
  • The 12 tickets have a reply, and support has a line for the rest.
  • Run the same count again in the morning, in case an event was missed.

Fix.

  • Test date code with a frozen clock on 28 and 29 February, 1 March and 31 December.
  • Alert when a Stripe payment has no matching subscription after a few minutes.

Follow-ups.

  • Tell finance and support what end date these customers got.
  • Note the date next year when these plans end, and check the renewal then.

Concept guide

Related questions