N+1 queries after ORM change

On-call incidentMedium35 min

Your task

Right after this morning's deploy, one page, the customer order-history view, starts timing out and the DB query count per request explodes. APM traces show a single page load now firing 800+ tiny near-identical SELECT * FROM line_items WHERE order_id = $1 queries instead of the one query it used to run, and the DB's query throughput chart spikes 50x while each individual query is fast (sub-millisecond). DB CPU is climbing from sheer query volume and connection churn. The deploy included an ORM change to how orders and their line items are loaded. Other pages are unaffected. Triage and fix.

It covers

  • Statement timeout
  • N plus one