What Is Minimum Detectable Effect in Incrementality Testing?
Minimum detectable effect (MDE) is the smallest lift your test could reliably have found. If your holdout came back flat and the minimum detectable effect was larger than the lift you would have acted on, the test did not find nothing. It was incapable of finding anything. Those are different results, and they point your budget in opposite directions.
What is minimum detectable effect, in plain English?
Plain version: your test can only catch a lift above a certain size, set by how much your sales move on their own. If that floor is 15% and the real lift was 8%, the test reads flat, not because the channel did nothing, but because 8% was always beneath what it could see.
Every test has this floor. Most readouts never tell you what it was.
How much your sales move on their own is what sets that floor. It is everything shifting your revenue that has nothing to do with your ads: promos, weather, a competitor's sale, a stockout in Georgia. The more your sales move, the bigger the lift has to be before the test can pick it out.
Here is what makes this expensive rather than merely technical. A flat result and a blind test produce the same word on the slide. Not significant. Somebody in that meeting is about to move budget on a sentence that was never finished.
So the question you take into the readout has three parts, and they have to be asked in order.
What was the ex-ante MDE, the ex-post MDE, and was the test powered for this decision?
That is question seven of the nine we think every measurement vendor should be able to answer. Ex-ante is what the design promised before the test ran. Ex-post is what the test actually delivered once the real movement showed up. And the third clause is the one that decides whether either number mattered.
One gate sits upstream of all three. Your test compares real markets against a pretend version of your business where the ads never ran. That pretend version never happened, so a model builds it, and statisticians call it the counterfactual. If that model could not have predicted your markets before the test began, none of the MDE questions matter. A precise answer to the wrong question is still wrong. Ask about the counterfactual first, then the MDE.
What was the ex-ante minimum detectable effect?
Calculated before the test from the movement you expect to face, the ex-ante MDE is the smallest lift this design should be able to spot. It is a planning number, and it is the only one you can still act on, because once the test is running the design is fixed.
Ex-ante MDE is where a test is won or lost, and almost nobody treats it that way. It gets computed, cleared, and forgotten. But the levers that set it are all pre-launch levers.
For a geo test those levers are the number of markets, how much they vary against each other, the size of your spend change, and how long you run. Baseline volume matters too, since a market with thin daily conversions moves around more from day to day.
What does not help is the lever A/B testing trained you to pull. You cannot buy more markets. There are roughly 210 in the country and that is the entire supply. Adding customers does not sharpen a geo test the way adding users sharpens an A/B test. Most sample size calculators marketers find are built for user-level A/B tests, not geo experiments. They assume the thing you can buy is more independent observations, and in geo that is usually not true.
One trap while you are still in planning. A single blended MDE across all your markets hides everything that matters. The same channel might deliver 15% incremental lift in one region and 3% in another. An average describes neither. Tests carrying real budget consequences get designed market by market, channel by channel.
If a vendor cannot tell you the ex-ante MDE before the test launches, they have not designed a test. They have scheduled one.
What was the ex-post minimum detectable effect?
Calculated after the test from the movement that actually showed up, the ex-post MDE is the smallest lift the test could really have caught. The ex-ante number describes the test you designed. The ex-post number describes the test you got. They are often not the same.
You plan against a calm stretch of history. Then a competitor launches, a promo moves a week, and a warehouse goes down for four days. All of that makes your sales move around more than you budgeted for, and that pushes up the smallest lift the test can catch.
Say the design promised you could catch an 8% lift. The extra movement meant it could really only catch 14%. If the test then comes back with a 6% estimate and no significance, the design looks fine on paper. But the test was blind in exactly the range where the answer was sitting.
That is the number the budget meeting needs. Most readouts stop one number too early.
One thing to be clear about: the ex-post MDE is not a hard cutoff. A test is built to catch a lift at that level about four times in five, and miss it one time in five. So even clearing the number is not a guarantee, which is why you read the whole range around the estimate, not just the threshold.
That range is the confidence interval, the band your true lift most likely sits inside. A good readout gives you both numbers. The ex-ante number tells you the design was sound. The ex-post number tells you whether the test you actually ran can support the call you are about to make.
Was the test powered for this decision?
This is the part that actually decides whether the test was worth running. A small MDE is not the point on its own. What matters is whether the MDE sits below the lift that would have changed what you do. Name that lift first, call it your decision line, then ask whether the test could ever have seen it.
Call it decision-powered testing. Whether a test is strong enough is not a fact about the experiment, it is a fact about the decision you ran it to make.
Where does your decision line come from?
Most marketers do not have one. It is not a feeling, it is arithmetic, and it starts somewhere unexpected.
iROAS is a revenue multiple, not a profit one. Earning a dollar back for every dollar spent is a loss at any margin below 100%. So breakeven is not 1.0x. Breakeven is 1 divided by your contribution margin. At a 40% margin, you need 2.5x before the channel pays for itself.
Now turn that into lift, since lift is what the test reports. The channel has to earn back its own spend at your margin, so your decision line as a percentage lift is:
decision line = channel spend ÷ (contribution margin × baseline revenue in the test markets)
Work an example. Your test markets do $2M of revenue during the window. The channel spends $200K. Contribution margin is 40%. The channel has to generate $500K of incremental revenue to break even, and $500K against a $2M base is a 25% lift.
Two things to keep straight, or the number comes out wrong. Use spend and baseline revenue from the same markets and the same test window. If the holdout only withholds spend in selected markets, the numerator is the spend at risk in those markets, not the full national budget. And use contribution margin after variable costs, not gross margin.
Sit with that number, because it is the whole problem.
Most MDE examples use tidy 5% and 10% effects because they come from A/B testing, where a 3% lift on a checkout button is real money. Your decision line, derived from your own margin, is 25%. A test with an ex-post MDE of 12% can see that comfortably. A test with an MDE of 30% cannot, and a flat result from it tells you nothing at all.
Most geo tests are run underpowered against the decision that prompted them, and nobody notices, because nobody computed the decision line first.
Breakeven is a floor, not the answer. Your line sits above it if you want the channel to earn rather than tread water, below it if the channel does strategic work you would fund at a loss. Either way, set the number before the test runs, not after the result disappoints.
Reading the result against the line
Then be precise about which number does what. The MDE tells you what the test could see. The range tells you what it ruled out. Compare the range against your decision line. Use the MDE to understand why the range came back as wide as it did.
Four things can come back, and each has exactly one right answer.
| What came back | The range | Could the test see your line? | What it means | What to do |
|---|---|---|---|---|
| A clear win | Sits entirely above your line | Yes | Strong evidence the channel matters | Scale or protect spend |
| Flat, narrow range | Top of the range falls below your line | Yes | Some effect possible, none big enough to matter | Cut or reallocate |
| Flat, wide range | Top of the range climbs above your line | No | Too much movement to decide anything | Do not cut on this test |
| Looks good, wide range | Straddles your line | No | Directional signal, not a budget mandate | Redesign, extend, or segment |
Those two middle columns are not padding. They tell the same story two ways: what the test could detect, and what the business can rule out. Read whichever one your CFO reads.
A null result does not mean cut. A narrow null means cut. A wide null means your test failed the decision.
What a short window can and cannot tell you
A geo holdout runs three to six weeks. What it measures is the incremental revenue that appears inside that window. Channels with long consideration cycles, and channels doing brand work, deliver revenue that lands after the test has closed. A narrow null bounds the effect in the window you measured. It does not bound the annual one.
So the honest version of "cut" is cut, and say what you measured. If the channel is short-cycle direct response, the in-window read is close to the whole story. If it is not, a flat result tells you the channel is not paying for itself inside three weeks, which may be exactly the wrong question to have asked.
That is what a media mix model is for, and it is why the two methods belong together rather than in competition. A holdout gives you a causal read over a short horizon. A model gives you contribution over a long one, and the holdout is what keeps the model honest.
What a null readout has to contain
Five things, every time. The observed lift, which is what happened. The range, which is what is still plausible. The ex-ante MDE, which is what the design promised. The ex-post MDE, which is what the test resolved. And your decision line, without which the other four mean nothing.
Then a verdict, and only three are available.
Evidence of no meaningful effect, in this window
The counterfactual held, the range is narrow, its top sits below your decision line. Cut, and say what you measured.
Insufficient resolution to decide
The counterfactual held, but the test could never have spotted a lift the size of your decision line. It cannot answer your question. Do not let it.
Invalid counterfactual, do not use
The model that built your control markets could not predict them. Nothing downstream is readable, whatever the p-value says.
So a weak answer to question seven sounds like this. "The result was not significant."
A strong answer sounds like this. "Observed lift was 4%. The range ran from minus 9% to plus 17%. Given how much the markets moved, this test could not have spotted anything under 20%. Your decision line, at your margin and your spend, was 25%. So this test was never capable of answering the question you asked it. Do not cut on this read."
Listen for which one you get.
The short technical version
For the person on your team who wants the mechanics. The smallest detectable lift comes out of two settings every test has: how sure you want to be before calling a result real (we use 90%), and how reliably the test should catch a true lift of a given size (we design to catch it four times in five). Tighten either one and the smallest lift the test can catch goes up.
Tools like Meta's GeoLift estimate the same idea through simulation: inject lifts of different sizes, rerun the design, and count how often the test detects them. We stay at 90% rather than a stricter bar because a marketing budget gets decided again every month, and being too cautious to ever act carries its own cost.
One more, if you run a lot of tests. If you are cutting many subgroups or running many experiments, treat isolated 90% wins with more skepticism, and decide your primary read before the test starts rather than after a surprising cut.
If a vendor cannot give you these numbers for your test, treat everything above as decoration.
How Stella runs this
We run measurement as a consultancy, not a dashboard you are left to read on your own. Every Stella readout ends in one of the three verdicts above, stated plainly: evidence of no meaningful effect, not enough resolution to decide, or an invalid counterfactual you should not use at all. You get the verdict and the reason behind it, including when the answer is that the test should not have been run, not a p-value to interpret yourself.
Every test gets the same check: a model of your control markets, tested against data it never saw, and a flag when the fit does not hold. In our benchmark of 225 geo tests, that check caught 10 that were not trustworthy and flagged 2 more that looked too perfect to believe. You can still launch. You just cannot say nobody told you.
Rather run it yourself? The self-serve platform uses the same check, plus a Location Selection Tool that finds the markets where the smallest detectable lift lands below your decision line, instead of letting you find out afterward that it never did. Sometimes the honest answer is that no set of markets gets you there, which is one version of when a test is the wrong call.
A flat result is only worth acting on when the control markets are believable, the test could see the lift that would change your mind, and the range rules out the move you were weighing. Most readouts will not show you that. Ask anyway.
A null result does not mean cut. A narrow null means cut. A wide null means your test failed the decision.
This is question seven of nine. The other eight cover the rest of what separates a number you can move budget on from one you cannot. Get the practitioner guide.
Frequently asked questions
What is minimum detectable effect in an incrementality test?
The smallest lift the test could reliably have found, given how much your markets moved on their own. A test with a 20% minimum detectable effect cannot tell you anything useful about a 12% lift, whatever its p-value says.
How do I know what lift is big enough to matter?
Derive it from your margin. Breakeven iROAS is 1 divided by your contribution margin, so a 40% margin needs 2.5x. As a lift, that is your channel spend divided by margin times the baseline revenue of your test markets. Spend $200K against a $2M base at 40% margin and your decision line is a 25% lift.
Does an inconclusive incrementality test mean the channel does not work?
No. It means the test did not find a lift big enough to clear the bar for calling a result real. Whether that is because the channel did nothing, or because the test was never sharp enough to see it, depends on the minimum detectable effect and the range around your estimate. Without both, you cannot tell the two apart.
What is the difference between ex-ante and ex-post minimum detectable effect?
Ex-ante MDE is worked out before the test from the movement you expect, and it tells you the design was sound. Ex-post MDE is worked out after the test from the movement that actually happened, and it tells you what the test could really catch. A good readout gives you both.
Does a higher confidence level make an incrementality test better?
Not at the design stage. Raising the threshold from 90% to 99% makes the test stricter and less sensitive, inflating the minimum detectable effect by 37%. After the test, the confidence level a result clears does indicate evidence strength.