Batch job throughput collapse
Your task
A nightly batch pipeline that processes a queue of jobs from a Postgres jobs table used to finish in 20 minutes; since you scaled the workers from 4 to 40 it now takes 3 hours and barely uses CPU. Workers claim jobs with SELECT ... FROM jobs WHERE status='pending' ORDER BY created_at LIMIT 1 FOR UPDATE, then update the row. With more workers, throughput went DOWN. pg_stat_activity shows most workers blocked on Lock. Triage and mitigate.
It covers
- Serializable
- Advisory lock
- Throughput collapse