Batch job throughput collapse

On-call incidentHard35 min

Your task

A nightly batch pipeline that processes a queue of jobs from a Postgres jobs table used to finish in 20 minutes; since you scaled the workers from 4 to 40 it now takes 3 hours and barely uses CPU. Workers claim jobs with SELECT ... FROM jobs WHERE status='pending' ORDER BY created_at LIMIT 1 FOR UPDATE, then update the row. With more workers, throughput went DOWN. pg_stat_activity shows most workers blocked on Lock. Triage and mitigate.

It covers

  • Serializable
  • Advisory lock
  • Throughput collapse