Invoice emails arrive hours late at month start
Your task
It is 08:40 UTC on the first of the month. Every customer's invoice is made today, and the invoice team promises them by 09:00. You are paged:
InvoiceQueueAgeHigh: oldest message in invoice-render is 2h 40m (threshold 30m)
The horizontal pod autoscaler (HPA) for the invoice workers shows:
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE invoice-worker Deployment/invoice-worker cpu: 94%/70% 2 6 6 2y
What else you see:
- At 06:00 the monthly run put 410,000 jobs on
invoice-render. The queue now holds 266,000 and falls by about 54,000 an hour. - Worker logs show no errors. Each pod renders about 2.5 PDFs a second.
- Cluster nodes are 45% allocated. The Postgres primary is at 30% CPU, and the email provider accepts mail as fast as it arrives.
- Support has 40 tickets from customers asking where their invoice is.
What do you do now, and what do you change before next month?
It covers
- Autoscaling
- Kubernetes
- Queue backlog