Autoscaler sees stale metrics
Your task
During an evening ramp your service saturates and p99 blows out, but the autoscaler never reacts, the fleet stays flat at its current size for 12 minutes. The HPA/scaler shows desired = current the whole time and its last-seen metric value is a low, stale number. Meanwhile your own monitoring graphs (a different pipeline) clearly show CPU and RPS climbing. There was no deploy to the service; an hour earlier, though, the metrics/observability platform had a brief incident. How do you triage and mitigate?
It covers
- Stale metrics
- Monitoring pipeline
- Blind autoscaler