Web checkout stalls while servers report nothing wrong

On-call incidentEasy15 min

Your task

It is 11:25. Your shop takes orders on its website and in its mobile apps. The last release was on Tuesday, and today is Thursday. This alert has been firing since 11:22:

OrderRateLow: orders per minute 56% below this time last week for 15 minutes (threshold 30%)

What you see:

  • Web orders fell from about 70 a minute at 11:05 to about 3 by 11:08. App orders are steady at about 50 a minute.
  • API 5xx errors are at 0.1%, as usual. Customers still load the checkout page, but POST /orders from the website fell with the orders. Payment success on the orders that do arrive is normal.
  • Customers tell support the Place order button "does nothing" and shows no error.
  • The browser error tracker shows an error that first appeared at 11:05, on almost every checkout page load: TypeError: Cannot read properties of undefined (reading 'init'). Its top frame is https://cdn.helpchat.example/widget/loader.js:1:8841, and the next frame is checkout.js:212.
  • The help chat widget sits behind a feature flag, help_chat, which each page reads as it loads.

How do you get web orders moving in the next few minutes, and what do you change so this does not stop checkout again?

It covers

  • Browser errors
  • Checkout
  • Web frontend

About this question

Web checkout stalls while servers report nothing wrong

Browser errorsCheckoutWeb frontend

Model answer

Free on every account

First move: turn off the help_chat flag. Each page reads it as it loads, so new checkout pages stop calling the widget. Open checkout in a private window and place a test order. Then watch POST /orders from the website and the error tracker for a few minutes. Web orders should climb back toward 70 a minute.

If your test order still fails, the call at checkout.js:212 probably runs outside the flag. Then ship a one-line try/catch around it through your fastest release path.

What this does not fix: anyone with a checkout page already open needs to reload it, so give support a line to send. The Place order click never sent a request, so these customers were not charged. Chat is now off for everyone, so tell support that too. About 67 orders a minute were lost for about 20 minutes, around 1,300 so far. Tell the business lead, since those carts may be worth an email once orders are back.

Read the signals. Only the website is affected: app orders are steady, so the orders API, the database and the payment provider all work. API errors are flat while POST /orders from the website fell, so the requests never leave the browser. A server cannot count an error for a request it never gets. A button that "does nothing" and shows no error fits a click handler that was never attached.

The browser error began at 11:05, and your last release was on Tuesday. Its top frame is in the chat vendor's loader.js, and the next is your checkout.js:212. So your code called into the vendor's file, and that file threw. Its URL serves whatever the vendor publishes, so it can change with no release of yours. An exception nobody catches ends the running script. If checkout.js attaches the Place order handler after line 212, the code that attaches it never runs. To show the vendor's frame, the tracker needs the script tag to have crossorigin and the CDN to send CORS headers. A wrapper in your code or the tracker's that catches the error, reports it and throws it again also works. Without either it would log Script error. with no stack.

Orders took until 11:08 to fall because customers who loaded the page before 11:05 could still finish.

The 3 orders a minute are probably browsers where the widget never ran, for example behind an ad blocker, or that still hold an old copy of the file.

Other causes do not fit. Orders that do arrive pay normally, so the payment provider is fine. A release would show in your own deploy history, and there is none since Tuesday. A flag change on your side at 11:05 would look the same, so check the flag's change history. Turning it off is right either way.

YesNoWeb orders 70 to 3, apporders steadyServer errors up?Look at the API and itsdependenciesRequests never leave thebrowserNew browser error at11:05, top frame in thechat loaderTurn off help_chat, testa real orderTell support: reload thepageIsolate the widget, adda browser check

Confirm. Read checkout.js around line 212: the widget call comes before the code that attaches the button's handler. curl -sI https://cdn.helpchat.example/widget/loader.js may show a last-modified time near 11:05, if the CDN sends one. With the flag off, the error count falls and your test order goes through. Then ask the vendor what changed.

Fix.

  • Attach the Place order handler first, and wrap every call into a vendor's code in try/catch, so its failure cannot stop your script.
  • Take chat off the checkout page, or load it only after checkout is ready.
  • If the vendor offers a versioned URL, use it, or host a copy you update on purpose. A subresource integrity hash, which makes the browser refuse a changed file, only helps once your code copes with the widget being absent.
  • Run a browser test that goes through checkout and clicks Place order every few minutes, and alert when it fails.
  • Alert on new browser errors on the checkout page.

Follow-ups.

  • Split the order alert by web and app. Web orders fell about 96%, but the total fell only 56%, and the page came at 11:22, 17 minutes after the error began.
  • Review the other vendor scripts on the checkout page.

Concept guide

Related questions