Uptime check fails from one probe location

On-call incidentEasy15 min

Your task

At 07:10 the external uptime monitor pages you: "Shop homepage DOWN". The check runs every minute from five probe locations (Virginia, Oregon, Frankfurt, Singapore and Sydney) and alerts when any one location fails twice in a row. Only Sydney is failing, with connect timeout after 10s; the other four get HTTP 200 in about 300 ms. The CDN dashboard shows normal traffic and a normal error rate from Australia, including the Sydney edge, and real-user monitoring shows Australian page loads at their usual speed. The monitoring vendor's status page lists "degraded connectivity from our Sydney probe location" since 07:02. What do you do?

It covers

  • Synthetic monitoring
  • False positive
  • Alert quorum