Segmenting a metric move: mix shift & Simpson’s paradox

The blended number can fall while every single segment rises — if you don’t slice, you’ll debug the wrong thing.

The idea

When a top-line metric moves, the first job is to localise it: slice by platform, geography, cohort, and tier until the change lives in a few cells, not the whole board. Averages hide as much as they reveal.

Two traps hide inside the blend. Simpson’s paradox: the overall rate drops even though each segment improved, because traffic shifted toward a lower-converting segment. And the whale effect: one giant account drags the mean far from the typical user, so the mean and median disagree.

How it works

The blended rate is a weighted average of the segment rates — weighted by each segment’s share of traffic. Change the weights and the blend moves even if every rate holds. That is the whole trick behind Simpson’s paradox.

blended = share_desktop*rate_desktop + share_mobile*rate_mobile

Period A   70% desktop @ 3.0%  +  30% mobile @ 1.5%
           = 0.70*3.0 + 0.30*1.5 = 2.10 + 0.45 = 2.55%

Period B   both rates UP, but mix shifted to mobile
           40% desktop @ 3.2%  +  60% mobile @ 1.8%
           = 0.40*3.2 + 0.60*1.8 = 1.28 + 1.08 = 2.36%   (down!)

Hold the mix at A's 70/30, keep B's higher rates:
           = 0.70*3.2 + 0.30*1.8 = 2.24 + 0.54 = 2.78%   (up)
           -> the drop was the MIX, not the rates.

For the whale trap the fix is just as simple: report the median alongside the mean. The mean is sum / count, so one huge value moves it; the median is the middle value, so a single outlier barely touches it. When they diverge, a few heavy users are carrying the average.

When to use it

Slice a metric when…	The trade-off
A top-line number moved and you don’t yet know where.	Slice too finely and every cell is noisy; start with a few big cuts.
Traffic composition could have shifted (a campaign, a new market, seasonality).	You must check the mix, not just the rates — easy to forget.
A spend or usage average looks surprising.	Mean and median together tell the story; either alone can mislead.

Watch out for

• Reading the blend as if the mix were fixed. A “conversion drop” is often just more traffic arriving in a lower-converting segment — the product didn’t get worse.
• Celebrating a blended rise for the same reason. The paradox runs both ways: the mix can flatter you while every segment quietly declines.
• Quoting the mean for skewed data. Revenue per user, session length, and spend are almost always right-skewed — the mean alone overstates the typical user.
• Slicing on a dimension caused by the change. If the treatment moved users between segments, segmenting on the post-change segment can itself create a false paradox. Slice on pre-change attributes.
• Stopping at the first slice that looks flat. A move can hide until you cross two dimensions — platform × geography — even when each alone looks calm.

Worked example

Interview prompt: “Signup conversion fell 4% last month. The PM is worried the new form is worse. What do you check first?”

Strong answer: “Before touching the form, I’d slice. I’d pull conversion by platform, source, and country, for last month versus the month before. Very often the form is fine and the mix moved — say a paid campaign brought a flood of mobile, cold traffic that always converts lower, so the blend fell while desktop and mobile each held or improved. I’d confirm by holding the traffic mix constant and re-blending: if the constant-mix rate is flat or up, the drop is composition, not the form. If instead a specific segment cratered, that’s where I’d dig.” That answer shows you localise before you theorise — and that you know a blended number is a weighted average that the mix can move on its own.

Check yourself