Pager fires for staging traffic

On-call incidentEasy15 min

Your task

At 02:47 you are paged: "Payments API error rate 100% (critical)". The production payments dashboard looks normal: error rate 0.2%, normal latency, and successful payments flowing at the usual night-time rate. The notification lists the firing series as:

{service="payments-api", env="staging", cluster="stg-eu-1"}

The staging calendar shows the platform team tearing down and rebuilding stg-eu-1 tonight between 02:30 and 04:00. The alert rule, routed to the production pager, is:

sum by (service, env, cluster) (rate(http_requests_total{service="payments-api",code=~"5.."}[5m]))
  / sum by (service, env, cluster) (rate(http_requests_total{service="payments-api"}[5m]))
  > 0.05

What do you do now, and what do you change tomorrow?

It covers

  • Alert rules
  • Label filters
  • False page