Signal vs noise: is this spike real?

Every process breathes within a band of ordinary noise — the skill is knowing when a bad day has actually left it.

The idea

Numbers move every day even when nothing changed. That ordinary wobble is common-cause variation — the natural noise of a stable process. Reacting to it (a stern email after one bad Tuesday) is called tampering, and it usually makes things worse.

A special cause is different: a real shift you can trace to a reason. A control chart tells them apart. You set a center line and limits at plus/minus three sigma from a stable baseline, then watch. Inside the band: noise, leave it. Outside the band, or a long run on one side: a signal, go investigate.

How it works

The limits come from a stable baseline — a stretch when nothing special was happening. Freeze them, then judge new days against them:

baseline mean (center line) .......... CL = 12 complaints/day
baseline spread .................... σ  =  3
upper limit .................. UCL = 12 + 3×3 = 21
lower limit .................. LCL = 12 - 3×3 =  3

signal rules (either one is enough):
  1. any point above 21 or below 3          -> special cause
  2. eight days in a row on one side of 12   -> special cause
otherwise -> common cause. leave the process alone.

Why three sigma? For stable data, only about 0.3% of days fall outside the band by chance — so a point outside is far more likely a real change than a fluke. A run of eight on one side has odds near 1 in 250, so it flags a quiet drift the single-point rule misses.

Isolating a change when several things moved at once: compare the recent window to the frozen baseline, and hold the confounds still. If you changed the return policy the same week a holiday hit, you can’t credit the policy until you compare like weeks (year-over-year, or a region that didn’t get the change).

When to use it

Watch out for

• Tampering. Adjusting the process after one point inside the band adds variation — you’re chasing noise. If it’s in the band, do nothing.
• Recomputing limits with the shift included. If you fold the bad week into the baseline, the widened σ swallows the signal and the chart goes quiet. Freeze limits from a clean period.
• Calling a lucky streak a trend. Three or four high days that never cross the limit are exactly what noise looks like — the band is there so you don’t over-read them.
• Ignoring the run rule. A small, real drift may never break a single limit but shows as eight-plus days on one side. Don’t wait for a dramatic spike.
• Crediting a change while a confound moved too. A holiday, a launch, a competitor outage the same week. Isolate the effect against a matched comparison before you claim the win.

Worked example

An interviewer says: “Complaints jumped 30% on Tuesday — what do you do?” The strong answer resists the reflex. First, plot it: is Tuesday outside the ±3σ band you’d built from a stable stretch, or just a loud day inside it? If it’s inside, it’s common cause — you note it and don’t reorganise the team over noise. If it’s outside or part of a run, it’s a special cause worth a root-cause hunt. And before you blame the new checkout flow that shipped Monday, you check what else moved — a promo, an outage, a holiday — and compare against a window where only the checkout changed. That sequence, baseline → compare → control for confounds, is what separates a calm operator from a fire-fighter.