Orders API jumps to 40 pods on a deploy
Your task
It is 14:10. The orders API runs on Kubernetes, about 10 pods at 500 requests a second. A horizontal pod autoscaler (HPA) sizes it from a Prometheus metric, aiming for 50 requests a second per pod, between 6 and 40 pods. Each pod opens 10 database connections when it starts. A deploy began at 14:04. This alert fires:
PostgresConnectionsHigh: connections above 90% of max_connections for 5 minutes (current 100%)
What you see:
kubectl get hpa ordersshows 40 replicas, its maximum. The load balancer counts about 500 requests a second, as it did at 13:30.- The HPA's metric is
rate(orders:http_requests:sum[2m]). The recording rule behind it issum(http_requests_total{app="orders"}), evaluated every 30 seconds. The HPA moved from CPU to this metric yesterday afternoon. - In Prometheus the metric read about 500 until 14:04. Between 14:04 and 14:07 it jumped to between 110,000 and 720,000 several times. Since 14:07 it reads about 500 again.
- Postgres logs
FATAL: sorry, too many clients alreadyfor new orders pods and for the inventory service.
What do you do in the next 10 minutes, and what do you change so deploys stop scaling the API?
It covers
- Prometheus
- Kubernetes autoscaling
- Recording rules