Pager fires for staging traffic
Your task
At 02:47 you are paged: "Payments API error rate 100% (critical)". The production payments dashboard looks normal: error rate 0.2%, normal latency, and successful payments flowing at the usual night-time rate. The notification lists the firing series as:
{service="payments-api", env="staging", cluster="stg-eu-1"}
The staging calendar shows the platform team tearing down and rebuilding stg-eu-1 tonight between 02:30 and 04:00. The alert rule, routed to the production pager, is:
sum by (service, env, cluster) (rate(http_requests_total{service="payments-api",code=~"5.."}[5m]))
/ sum by (service, env, cluster) (rate(http_requests_total{service="payments-api"}[5m]))
> 0.05
What do you do now, and what do you change tomorrow?
It covers
- Alert rules
- Label filters
- False page