Month-end finance reports fail partway through

On-call incidentEasy15 min

Your task

It is 08:40 on the first working day of the month. The finance team's reporting tool reads from pg-reports, a Postgres 16 streaming replica of the main database, so heavy queries stay off the primary. Only the reporting tool uses pg-reports. Finance needs the month-end numbers by 12:00. This alert fires:

ReportFailures: 23 report runs failed since 08:00 (threshold 5)

What you see:

  • Each failed run ends with ERROR: canceling statement due to conflict with recovery and DETAIL: User query might have needed to see row versions that must be removed.
  • Runs fail between 30 seconds and 4 minutes after they start. Reports that finish within 30 seconds never fail. Today's month-end reports take 2 to 6 minutes each.
  • Last month the same reports took about 25 seconds. Since then finance added 3 years of history to them.
  • pg-reports is less than 1 second behind the primary, and its CPU is at 40%. On the primary, orders and payments get about 300 updates a second in business hours, and autovacuum cleans each of them every few minutes.

What do you change so finance has its numbers by 12:00, and what could that change cost you later?

It covers

  • Hot standby
  • Query conflicts
  • Postgres