Error alert reports no data after a library upgrade
Your task
At 14:00 you get a low-urgency notification: the Grafana alert rule "Checkout 5xx rate above 2%" is in state NoData, and has been since 11:40. The checkout dashboard's request-rate panel is also empty since 11:40, but the order count in the database is normal for a Tuesday. At 11:35 checkout deployed a change that upgraded its HTTP instrumentation library to a new major version. The alert query is:
sum(rate(http_requests_total{service="checkout",status=~"5.."}[5m]))
/ sum(rate(http_requests_total{service="checkout"}[5m]))
In Prometheus the checkout scrape target is UP, and /metrics on a checkout pod now shows lines like http_server_requests_total{service="checkout",code="200"} 48211. What do you do?
It covers
- Alert rules
- Metric rename
- No data