Customers get overdue notices for invoices not yet due
Your task
It is 10:20. Your invoicing product emails a reminder when an invoice is overdue. This alert has been firing since 10:12:
SupportTicketSurge: 90 billing tickets in the last 30 minutes (threshold 25)
What you see:
- Customers forward an email that says "Invoice 88213 is overdue". The name and invoice number are real, but many of these invoices are not due for weeks. The Pay now link goes to
https://staging.billing.example/pay/88213. - Production's reminder job last ran at 06:00 and sent 412 emails. Its next run is tomorrow at 06:00.
- The email vendor's activity log shows 40,000 emails since 10:00, at about 2,000 a minute, all sent with the API key
billing-staging. - At 09:50 Marek refreshed the staging database from last night's production backup, ahead of a sales demo. Staging runs its reminder job every hour, on the hour, and testers set it to treat every open invoice as overdue.
What do you stop first, and what do the people who got this email need from you today?
It covers
- Bulk email
- Test environments
- Customer communication
About this question
Customers get overdue notices for invoices not yet due
Model answer
Free on every accountFirst move: disable the billing-staging API key at the email vendor. That stops every send at once, wherever it comes from, and it needs no access to staging. Its cost: anything else on this key stops too. The name says only staging uses it. Check the vendor's log for other mail on it, but do not delay for that. If your vendor has no disable, revoke the key. Then stop staging's reminder job, because it runs again at 11:00 and could start over if someone turns the key back on for the demo. Stopping the job first is fine if you can do it in seconds, as long as the running batch dies with it.
What this does not fix: the 40,000 emails already sent, and any the vendor has accepted but not yet delivered, so check its queue and cancel those if it lets you. Real customer data is also still in staging, and the links are still live.
The link. With 40,000 emails out, some customers will click Pay now. Take /pay/* on staging offline, point it at a plain notice, or redirect it to production's pay page for the same invoice, so nobody pays through a test system. Read staging's logs for anyone who submitted a payment. They may believe they have paid, so they get a personal reply and a check by finance. Tell Marek that staging is out of the demo until it is clean. Tell your security or privacy lead that production data sits in staging, since they decide whether anyone must be notified.
What the people need. Give support a reply to use now. Then:
- Export the list from the vendor's log: this key, since 10:00. Do it after sending stops, so the list is final.
- Send one correction from production to exactly that list. It says the notice came from a test system by mistake, not to use the link, and gives each invoice's real due date.
- Merge each message with the invoice's real status. Some recipients really are overdue, including any of the 412 from 06:00 that staging reached, and they must not be told they owe nothing. The backup is from last night too, so some have paid since.
Read the signals. The link points at staging. Production sent 412 emails at 06:00, not 40,000, and it does not run again until tomorrow. The vendor's log ties all 40,000 to billing-staging, at 2,000 a minute for 20 minutes. The first email went at 10:00, the first hourly run after Marek's 09:50 refresh. Until then staging held no real customers, or an earlier run would have mailed them. The restore brought in real customers and real email addresses, and the testers' setting explains why invoices not yet due were chased.
Other causes do not fit. A stolen key would not explain real invoice numbers with staging's link, or the timing against the refresh. The vendor's log may also show where the calls came from.
Confirm. The sent count for billing-staging stops rising in the vendor's log. A sample of staging's customer table holds real addresses. Staging's job log shows a run that began at 10:00.
Fix.
- Have the refresh replace every email address and phone number with a test one before any app or job starts. Mask other personal data too, or use made-up data.
- Give staging a sender that cannot reach customers even if its data goes wrong: a sandbox server or domain that delivers to nobody or only to listed addresses (Postmark and Mailgun have these), or on Amazon SES a key whose policy limits
ses:Recipients. A test flag set per request, such as SendGrid's sandbox mode, holds only if staging's code sets it. - Leave scheduled jobs off after a refresh until a person turns them on.
- Alert on send volume per key. A staging key should send a handful.
Follow-ups.
- Check what else staging can reach: text messages, webhooks and payment keys.
- Review who may restore production backups into staging.