Attribution models vs incrementality

Attribution decides how to split the credit for a sale. Only a holdout can tell you whether the channel actually caused it.

The idea

A customer touches several channels before buying — social, then email, then a branded search click. Attribution models are just rules for slicing that one sale’s credit among the touches. First-touch gives it all to social; last-touch gives it all to search; linear splits evenly; position-based favors the ends; data-driven learns a weighting from the data.

But every one of them is dividing a pie that already exists. None of them answers the real question: if we turned this channel off, would the sale still have happened? That’s incrementality — and you can only measure it by running an experiment: withhold the channel from half the audience (a holdout) and see how much conversions actually drop.

How it works

Attribution partitions one observed conversion. Incrementality estimates the counterfactual — what would have happened without the channel — which no attribution rule can see from the data it has.

Journey:  Social -> Email -> Search -> Purchase   (one sale, split 100%)

  first-touch    Social 100 · Email  0 · Search   0
  last-touch     Social   0 · Email  0 · Search 100
  linear         Social  33 · Email 33 · Search  33
  position (U)   Social  40 · Email 20 · Search  40
  data-driven    Social  56 · Email 33 · Search  11

Holdout:  withhold a channel from half the audience.
  incremental lift = (conv_treatment - conv_holdout) / conv_treatment

  Social held out:  8.0% -> 5.6%   lift = 30%   (last-touch credited it 0%)
  Email  held out:  8.0% -> 6.6%   lift = 18%
  Search held out:  8.0% -> 7.5%   lift =  6%   (last-touch credited it 100%)

Read those last two lines together. Last-touch says Search drove the whole sale; the holdout says turning Search off barely moves conversions — it was capturing demand social and email already created. That gap between credit and lift is exactly what you’d over-invest in if you trusted attribution alone.

When to use it

Watch out for

• Treating last-touch as truth. It systematically over-credits bottom-funnel channels — branded search, retargeting — that mostly harvest demand others created.
• Calling data-driven attribution ‘causal.’ It’s a smarter split of observed paths, not an experiment. It still can’t see the counterfactual; only a holdout or MMM can.
• Reading a holdout with no confidence interval. A small test with noisy conversions can show “lift” that’s just chance. Size the test and report the interval, not a point estimate.
• Double-counting across channels. Each team’s last-touch report can sum to more than 100% of real conversions. Incremental lifts don’t have to sum to 100% — and usually don’t.
• Cutting a ‘low-credit’ upper-funnel channel. Kill social because last-touch shows it at 0%, and paid search may quietly dry up too — you removed the demand it was harvesting.

Worked example

An interviewer asks: “Branded search shows the best ROAS in our last-touch report. Should we put more budget there?” The disciplined answer separates credit from cause: “Last-touch will always flatter branded search because it’s the final click before purchase — but many of those people were already going to buy. Before shifting budget I’d run a geo or audience holdout on branded search and measure the incremental lift. If turning it off only drops conversions a few percent, its true value is far below its attributed credit, and the marginal budget is better spent upper-funnel — where a holdout would likely show higher lift.” Naming the holdout as the tie-breaker, not a prettier attribution model, is what signals you understand causation.