The signal that is too small to prove

A signal you cannot prove is still information — the skill is buying just enough of the right evidence, cheaply and fast, to know what to do next.

The idea

Most of the things that matter arrive small: a two-point drop, three customers saying the same odd sentence, an engineer who has gone quiet, a dashboard that is green while the room is not. None of it is significant. All of it is early.

The mistake on one side is dismissing it because the evidence is thin. The mistake on the other side is a reorganisation on Monday. Both are ways of avoiding the actual work, which is modest and quite ordinary: turn the hunch into a claim that could be wrong, buy the cheapest observation that would separate the worrying explanation from the innocent one, respond in proportion to what you now know, and write down in advance the number or the date at which this stops being a hunch and becomes an escalation.

Time is not free while you decide. If the cause is real, harm accrues every day you spend being careful. That is the tension the simulation below lets you feel.

Buy the cheapest observation that could change your mind

Set your tripwire first — before you know anything

Activation rate fell from 62.1% to 60.1% in the week of the 4 March release. You do not yet know why. Pre-commit the rule that turns this into someone else's problem, so it is decided once rather than argued about every Monday.

Escalate if the post-change gap reaches 6 points

…or by day 10, whichever comes first

The same two-point wobble sits under both hidden causes. After you close a run, replay it against the other cause and watch the identical cohort split mean something completely different.

How it works

Four moves, in order. Each one is small enough to do before lunch.

Convert the hunch into a claim that could be wrong. "Something feels off" cannot be investigated. "Accounts onboarded after the 4 March change activate at a lower rate than accounts onboarded before it" can — and so can its innocent twin, "the 4 March analytics swap under-counts activation." Write both.
Pick the cheapest observation that separates them, and say how long it takes. An observation that reads the same under both explanations is not evidence, however expensive it was.
Respond in proportion to what is known. Almost never a reorganisation. Almost always a question asked of a named person, with a date attached.
Set the tripwire before you start — a number or a date at which this becomes an escalation — so the decision is made once instead of re-litigated weekly.

Signal: activation 62.1% -> 60.1% (-2.0 points), week of 4 March.

Claim (could be wrong):
  "Accounts onboarded after the 4 March release activate less
   than accounts onboarded before it."
Innocent twin:
  "The 4 March analytics swap under-counts the activation event."

Candidate observations         cost            separates the two?
  ask Dana (owns onboarding)   1 day,  1 hr    yes
  read the 4 March deploy log  1 day,  2 hrs   yes
  reconcile vs server-side     2 days, 3 hrs   yes
  split metric by cohort       2 days, 4 hrs   NO  - identical either way
  call three customers         4 days, 6 hrs   yes
  wait another week            7 days, 0 hrs   yes (step vs slope)

Cost of being careful, if the cause is real:
  harm(d) = d^2/18 + d/2        accounts lost by day d
  harm(1)  =  0.6     harm(7)  =  6.2
  harm(14) = 17.9     harm(21) = 35.0     harm(35) = 85.6

  ask Dana today   ->  1 day  ->  ~1 account
  wait one week    ->  7 days ->  ~6 accounts
  wait a month     -> 28 days -> ~58 accounts

Tripwire, written down on day 0:
  "If the post-change cohort gap reaches 6 points, or it is still
   there on 18 March, it goes to the VP of product. Until then it
   is Dana's, and she reports on Friday."

Note the shape of that table: the most expensive observation on the list is the one that separates nothing, and the cheapest is a one-day question to a person who already knows. That inversion is common, and it is why "run the analysis" is so often the wrong first instinct.

When to use it

Situation	Fit
An early metric move that is inside noise, but arrives right after a known change	Strong — the change gives you a claim to test and a deploy log to read.
Repeated qualitative signal: three customers, two engineers, the same sentence	Strong — repetition from independent sources is weak evidence that is cheap to sharpen.
A person signal: someone gone quiet, a team's tone changed	Strong — the separating observation is a private conversation, cost one hour.
Slow-moving, high-frequency metrics with a real experiment available	Weak — if you can run a proper test cheaply, run it; do not reason from vibes.
Safety, security, or anything irreversible	Different rules — escalate on the weak signal itself. Asymmetric downside beats cost efficiency.

The trade-off. You are deliberately buying a partial answer. You will sometimes act on a signal that turns out to be nothing, and you will occasionally close one that turns out to be something. The method does not remove those errors; it makes them cheap, fast, and visible — and it stops the same argument being had every week.

Watch out for

Treating "not statistically significant" as "nothing is happening." Significance is a statement about your evidence, not about the world. The right response to a thin signal is a cheap observation, not dismissal.
Buying evidence that cannot separate the explanations. The cohort split in the simulation reads exactly the same whether onboarding broke or the analytics client did. Before you spend, ask: what would this look like if I am wrong? If the answer is "the same", do not buy it.
The response that manufactures the problem. A war room, a paused roadmap and a daily retention standup, bought with one ambiguous chart, costs more than the churn you feared — and quietly teaches everyone that raising a small signal is dangerous. You will get fewer signals next quarter.
The tripwire that is a feeling rather than a number. "We'll keep an eye on it" is a promise to re-litigate weekly. "Six points, or the 18th, whichever comes first, then it goes to Priya" is a decision made once.
Letting the tripwire drift when it fires. The whole value is pre-commitment. If the number is hit and you renegotiate the number, you never had a tripwire — you had an opinion.
Investigating instead of asking. Analysts are slower than people. The person who shipped the change on the 4th usually knows in one sentence what a fortnight of segmentation would have suggested.

Worked example

An interviewer says: "Your weekly activation rate dropped two points last week. Your data scientist says it is not significant. What do you do?"

A strong answer does not reach for a bigger sample. It says: I would first write the claim so it could be wrong — accounts onboarded after the 4 March release activate less than those onboarded before it — alongside the innocent version, the release swapped the analytics client and we are under-counting. Then I would buy the cheapest observation that tells those apart, which is not an analysis: it is one message to Dana, who owns onboarding, and ten minutes in the 4 March deploy log. That is a day, and about three analyst-hours. A cohort split, which is what everyone asks for first, would cost twice as much and would look identical under both stories.

Then the response, sized to what I actually know: Dana owns it, she reports Friday, nobody's roadmap moves. And the tripwire, said out loud in the same breath: if the post-change cohort gap reaches six points, or it is still there on the 18th, it goes to the VP and we stop shipping onboarding changes. If the cause turns out to be instrumentation, I also restate the last three weeks so nobody plans off a false drop. That answer shows the interviewer four things at once: you can make an ambiguous thing testable, you can price evidence, you can size a response, and you can pre-commit rather than agonise.