Hundreds of orders go through at zero dollars

On-call incidentEasy15 min

Your task

It is 10:05 on a Tuesday. Finance posts in the on-call channel: revenue this morning is far below normal, but order volume looks fine. You check:

  • Since 09:30, 412 orders have a total of $0.00. A normal morning has 2 or 3.
  • 409 of those 412 orders used the promo code SPRING10. Marketing emailed it to newsletter subscribers at 09:00 as 10% off.
  • The promotions admin shows that SPRING10 was last edited at 09:27. No code has been deployed since yesterday.
  • A thread on a deals forum titled "free stuff here" links to the shop and has 300 replies.
  • The warehouse picks the morning's orders for same-day shipping at 13:00.

What do you do in the next 30 minutes, and what do you change afterwards?

It covers

  • Blast radius
  • Config change
  • Stop the bleeding