Invoice emails arrive hours late at month start

On-call incidentEasy15 min

Your task

It is 08:40 UTC on the first of the month. Every customer's invoice is made today, and the invoice team promises them by 09:00. You are paged:

InvoiceQueueAgeHigh: oldest message in invoice-render is 2h 40m (threshold 30m)

The horizontal pod autoscaler (HPA) for the invoice workers shows:

NAME             REFERENCE                   TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
invoice-worker   Deployment/invoice-worker   cpu: 94%/70%   2         6         6          2y

What else you see:

  • At 06:00 the monthly run put 410,000 jobs on invoice-render. The queue now holds 266,000 and falls by about 54,000 an hour.
  • Worker logs show no errors. Each pod renders about 2.5 PDFs a second.
  • Cluster nodes are 45% allocated. The Postgres primary is at 30% CPU, and the email provider accepts mail as fast as it arrives.
  • Support has 40 tickets from customers asking where their invoice is.

What do you do now, and what do you change before next month?

It covers

  • Autoscaling
  • Kubernetes
  • Queue backlog