Weighing policy evidence: validity and transferability

A pristine study of somewhere else can tell you less about here than a messy study of home.

The idea

The evidence hierarchy ranks designs — randomized trial over quasi-experiment over observational over qualitative. Useful, but it answers only one question: internal validity, how believable the estimate is for the setting it came from. It says nothing about transferability — whether that estimate travels to your population, prices, and institutions.

So a distant randomized trial can deserve less weight than a nearby weaker study. The job isn’t to grab the top of the hierarchy; it’s to weigh each study by precision × how much its setting looks like yours — and to keep effect size and statistical significance separate from that.

How it works

Grade each study on two axes, then pool. Weight is precision discounted by how far the setting is from yours:

Two axes per study
  internal validity  ->  precision of ITS estimate   (1 / variance)
  transferability    ->  how much its setting matches yours   (0..1)

Weight for predicting the LOCAL effect
  weight = transferability / variance
  (a tight foreign estimate with poor context match gets little weight)

Pool
  predicted = sum(weight * effect) / sum(weight)
  band width grows with 1/sum(weight)  +  a structural-uncertainty term
  qualitative evidence shrinks that structural term by grounding
  whether the mechanism operates here

Worked, with the defaults on screen
  foreign RCT   effect +12 pp,  variance 1.5,  transfer 0.43
  local obs.    effect  +5 pp,  variance 4.0,  transfer 0.90
  weights       12: 0.29   5: 0.23   ->  pooled ~ 8.9 pp
  drop "institutions" match and the local study out-weighs the RCT

Watch the separation the tool makes concrete. Each study’s own whisker (its 95% CI) can exclude zero — “significant” where it was run — while the pooled local band still includes zero. Significance in another setting is not evidence of a local effect.

When to use it

Weigh evidence this way when…	The limit
You’re importing a result from elsewhere to predict a local effect.	Transferability is a judgment, not a measured number — be explicit about it.
Studies disagree, or the strongest design comes from the least similar place.	Weights depend on variance and match estimates you may only know roughly.
A qualitative or mechanism study can tell you whether the causal story even holds here.	Qualitative work grounds transfer; it rarely sizes the effect on its own.

Watch out for

• Worshiping the hierarchy. “It’s an RCT” settles internal validity, not transferability. A randomized trial from a very different place can deserve real discounting.
• Confusing significance with size — or with relevance. A tiny, precisely-estimated effect can be significant and useless; a large effect from an unlike setting can be significant there and irrelevant here.
• Ignoring internal validity in local studies. Nearby isn’t enough — an observational local study still carries confounding, which is why it gets a wider own-CI, not a free pass.
• One-number transferability. Population, scale, and institutions can differ independently; a perfect population match won’t rescue a policy whose delivery institutions don’t exist locally.
• Throwing out qualitative evidence. It won’t give you an effect size, but it tells you whether the mechanism plausibly operates here — which is exactly what transfer turns on.

Worked example

An interviewer says: “A famous randomized trial abroad found a job-training program raised employment 12 points. Should we expect that here?” A senior answer refuses the shortcut. It credits the trial’s internal validity, then asks the transfer questions: is our unemployed population similar, is our labor market as slack, do we have the delivery institutions the trial relied on? If institutions differ sharply, the pristine estimate gets discounted — and a rougher local before-and-after study, plus interviews confirming the mechanism, may carry more weight for predicting our effect. The honest output is a range, wider than any single study’s, centered below 12 — and a note that the band still comfortably clears zero, so the direction is safe even if the magnitude isn’t.