Receipts and password resets stop reaching most customers

On-call incidentEasy15 min

Your task

It is 09:50. Your app sends about 30 receipts and password reset emails a minute. At 09:30 the team moved sending from the old email vendor to a new one by changing one setting, EMAIL_VENDOR, on the notifier service. The old vendor's account stays open until the end of the month. This alert has been firing since 09:36:

EmailBounceRate: hard bounces 74% over the last 15 minutes (threshold 5%)

What you see:

  • A hard bounce is a permanent refusal by the receiving mail server. Bounces began at 09:32, and before 09:30 they were under 1%.
  • The new vendor's log shows this for Gmail addresses: 550 5.7.26 Unauthenticated email from shop.example is not accepted due to domain's DMARC policy. Outlook addresses bounce with a similar line. DMARC is a DNS record that tells mail servers what to do with mail that fails sender checks, and shop.example publishes p=reject.
  • The migration checklist has 6 items, and 5 are ticked. The open one is "Add the vendor's 3 DNS records", assigned to Priya, who is on leave until Monday.
  • Password reset links last 30 minutes. notifier does not retry a bounced email.

How do you get mail flowing in the next 15 minutes, and what do you do about the messages that already bounced?

It covers

  • Email delivery
  • Dns
  • Vendor migration

About this question

Receipts and password resets stop reaching most customers

Email deliveryDnsVendor migration

Model answer

Free on every account

First move: set EMAIL_VENDOR back to the old vendor. Its account is still open, and its DNS records let its mail pass. First check that the 5 ticked checklist items did not remove them. The last change reached new sends after about 2 minutes, so expect the same. If sends have not moved by then, restart notifier, in case it reads the setting only at start, one instance at a time so sending does not stop. Send a test to a Gmail and an Outlook address, and watch new sends arrive. The alert's 15-minute figure lags, so judge by new sends.

Adding the 3 DNS records now is the other route. It is fine if you have DNS rights and the vendor can verify them within the 15 minutes. New names usually resolve within minutes, unless a recent failed lookup is still cached, and a typo costs you another round, or breaks the old vendor's mail if the change touches your SPF record. Do not relax DMARC to p=none while the old vendor works. It takes effect only as the record's old value expires from caches, and it lets anyone send mail as your domain.

What this does not fix: the messages that already bounced, and the new vendor's setup.

The messages that bounced. From 09:32 to 09:50 is 18 minutes, at 30 emails a minute, so 540 emails went out, and 74% of that is about 400 bounced so far. That grows by about 22 a minute until the switch back lands. notifier will not retry them.

  • Export the bounced addresses and message IDs from the new vendor's log. Its log may not hold the bodies, so rebuild each receipt from the order. The other 26% did not hard bounce. Check the log's delivered status, and leave delivered ones alone.
  • Resend receipts through whichever vendor now passes, once each, keyed on the order, so a rerun does not send them twice.
  • Do not resend password reset links wholesale. The first die at 10:02 and the last sent so far at about 10:20, so a resend arrives with little time left. Many of those customers have already asked again or given up. A link that turns up late also looks like phishing. Customers who still need one ask again, so tell support and put a line on the sign-in page if you can. Resending only links issued in the last few minutes is fair, but asking again is simpler.
  • Many vendors stop sending to an address after a hard bounce. Check the new vendor's suppression list and clear these addresses before that vendor sends to them again, or their mail will silently drop.

Read the signals. Bounces began 2 minutes after the switch, at addresses that worked at 09:29, so the addresses are fine. The vendor is up, and receivers answer. The 550 line names the reason. DMARC passes only when SPF (which servers may send for a domain) or DKIM (a signature on the message) passes for a domain matching the From address. The new vendor's DNS records are the open checklist item. They are usually a DKIM key and a return-path record, which sets the bounce address SPF checks. Without them, its mail cannot pass for shop.example, and p=reject tells receivers to refuse it. The 26% that did not bounce probably went to servers that do not enforce the policy. The alert first fired at 09:36, when about 20% of its 15-minute window had bounced. Since 09:47 the window holds only the new vendor's mail, so it reads the full 74%.

Yes, DMARCNo74% hard bounces since09:32Does the bounce name apolicy?New vendor's DNS recordsare missingLook at addresses andcontentSwitch back, or add therecords if they verifyin timeClear bounced addressesat the new vendorResend bounced receipts,not old reset linksFinish and verify therecordsTest at Gmail andOutlook, then move again

Confirm. dig +short TXT _dmarc.shop.example shows v=DMARC1; p=reject. The new vendor's dashboard lists the 3 records as unverified, and dig on the names it gives returns nothing. After the switch back, open a test message in Gmail with Show original. It shows dmarc=pass, with dkim=pass or spf=pass beside it.

Fix. Before the next move:

  • Give Priya's item to someone who is here, since the old account closes at the end of the month. Add the records and wait for the vendor to verify them.
  • Send tests through the new vendor to Gmail and Outlook, and read the authentication results.
  • Move a small share first if notifier can split traffic. If it cannot, move at a quiet hour with someone watching.

Follow-ups.

  • Make the DNS item a blocker on the migration checklist, so the switch cannot happen with it open.
  • Give notifier a way to resend a failed receipt.

Concept guide

Related questions