Month-end finance reports fail partway through
Your task
It is 08:40 on the first working day of the month. The finance team's reporting tool reads from pg-reports, a Postgres 16 streaming replica of the main database, so heavy queries stay off the primary. Only the reporting tool uses pg-reports. Finance needs the month-end numbers by 12:00. This alert fires:
ReportFailures: 23 report runs failed since 08:00 (threshold 5)
What you see:
- Each failed run ends with
ERROR: canceling statement due to conflict with recoveryandDETAIL: User query might have needed to see row versions that must be removed. - Runs fail between 30 seconds and 4 minutes after they start. Reports that finish within 30 seconds never fail. Today's month-end reports take 2 to 6 minutes each.
- Last month the same reports took about 25 seconds. Since then finance added 3 years of history to them.
pg-reportsis less than 1 second behind the primary, and its CPU is at 40%. On the primary,ordersandpaymentsget about 300 updates a second in business hours, and autovacuum cleans each of them every few minutes.
What do you change so finance has its numbers by 12:00, and what could that change cost you later?
It covers
- Hot standby
- Query conflicts
- Postgres