MMM Out-of-Sample Validation: What R-Squared Tells You
R-squared tells you how much of the movement in your sales a model can explain. In-sample R-squared is scored on the same data the model trained on, so it mostly reflects how well the model memorized its own history. Out-of-sample R-squared is scored on a stretch the model never saw, so it is the one that tells you whether the model predicts. The catch most teams miss: an out-of-sample R-squared that comes back suspiciously high is often not a strong model at all, it usually means a leak.
What is out-of-sample validation?
Out-of-sample validation holds back a chunk of your real history, refits the model on the rest, then predicts the held-out window and measures the miss. Read the miss as a percentage: predict 100K, actual was 88K, that is a 12% error. It is the check that tells you whether the model learned how your business works or just memorized its past, the difference between an answer key and a real exam.
Picture the usual deliverable. Six months and a real chunk of budget in, the consultant hands over a deck: channel-level ROIs, saturation curves, recommended reallocations, and an R-squared of 0.92. It looks strong, then it sits on a shelf because no one is sure it holds up.
That 0.92 is almost always in-sample, which measures how well the model explains the data it trained on. A high number there is expected, not reassuring. The number that should decide whether you move budget is the out-of-sample one, and it is the number most decks leave out. It is one of the checks we hand clients for grading a measurement vendor.
Why isn't a high R-squared the answer?
Because the high number on the deck is almost always the in-sample one, and in-sample fit rewards memorizing the past. The definition of R-squared does not change between the two versions, it is the share of the ups and downs in your sales the model accounts for. What changes is the data it is scored on, and on the data the model trained on, a high score says little about next quarter.
Even the out-of-sample number needs a second look before you trust it. If your revenue has strong seasonality and trend, a model can score high without knowing anything causal about your marketing, because the calendar alone predicts most of the movement. The check that settles it is whether the model beats a seasonal-naive baseline: does it predict the held-out weeks better than just repeating the same weeks last year, adjusted for growth? If it does not clear that bar, the R-squared is measuring the calendar, not your media.
Practitioners disagree on one point here, and the disagreement is legitimate. Google's Meridian documentation argues that out-of-sample prediction is not the right objective for a causal MMM, and that some overfitting is acceptable if the model includes the right confounders. We think that is right in principle and risky in practice. Trusting that a vendor included the right confounders, with no out-of-sample check to back it, is how unvalidated models end up steering budget. Prediction accuracy is not the goal, but it is the cheapest check that the model is not fooling itself, so we run it and read it with the seasonal-baseline caveat above.
How do you check an MMM for overfitting?
Compare the two R-squared numbers and read the distance between them. A model that scores high in-sample and drops sharply out-of-sample fit the training noise instead of the underlying pattern, and it will break as soon as it meets new data. So do not ask a vendor for the R-squared, ask for both, and look at the gap.
A model at 0.97 in-sample and 0.62 out-of-sample memorized its history. A model at 0.91 in-sample and 0.89 out-of-sample is generalizing. The figures below are illustrative, but the shape is what matters.
One thing trips up most write-ups on this. On a fair holdout, one that is long enough and seasonally representative, overfitting pushes the out-of-sample number down, it does not prop it up. So a high out-of-sample R-squared on a fair holdout is not itself overfitting. Something else is holding it up.
Two things can. The first is an easy test: a short window, an unusually stable stretch, or a business so seasonal the calendar does most of the predicting while the media coefficients underneath could be nonsense. The seasonal-baseline check above catches that one. The second hides inside a test that looks rigorous, and it is the one that costs you money: leakage.
What is leakage in an MMM?
Leakage is when a variable in your model is coupled to the outcome at the same point in time, so the model can partly see the answer inside a window that is supposed to be blind. To predict the held-out weeks, the model still gets fed its input variables for those weeks. If one of those inputs moves with revenue contemporaneously, branded search volume or site sessions in the same week you are trying to predict, the model reconstructs holdout revenue from the proxy instead of from anything causal. The window is technically held out and functionally wide open.
That is why leakage is the dangerous one. A soft test just flatters the fit. Leakage corrupts the coefficients. The model leans on the outcome-coupled variable and discounts the media channels you are trying to measure, so the R-squared looks incredible and the per-channel ROIs, the numbers you reallocate budget on, are quietly wrong. It is the failure mode where the model that scores best on the usual metrics is the one to trust least.
An out-of-sample R-squared that sails past the realistic band for MMM should prompt a question rather than a celebration: what in this model already knows the answer? The usual suspects, and the reason each is a problem:
| Variable | Why it moves with revenue | Why it corrupts the model |
|---|---|---|
| Branded search volume | People search your brand because they are already converting, so volume moves with revenue in the same week. | It is a demand signal, not a lever you pull. As a regressor it reads demand back to the model rather than measuring what your marketing did. |
| Site sessions or traffic | Sessions are downstream of the same demand that produces revenue. | Feeding them in lets the model shortcut straight to the outcome. |
| Affiliate spend | Affiliates get paid on conversions, so affiliate spend tracks sales almost by construction. | In a holdout, that is close to handing the model the test key. |
One caveat so this does not get misread. The problem is not branded search as a paid channel, plenty of brands run it deliberately and defend it. The problem is feeding the model a variable that moves with demand rather than one you set. Branded search spend you control and hold roughly flat behaves like a channel. Branded search volume, impressions, or sessions, which rise and fall with demand, are the leak. When in doubt, ask whether you set the number or your customers did.
The fix is not to interrogate the R-squared, it is to interrogate the feature list. That happens in two places. Before you fit, screen every variable with one question: would I know this number at the moment I am forecasting, and is it something my marketing changes, or something that changes with my sales? If it is the latter, it does not belong in the model.
After you fit, run the test that settles it. Take the suspect variable out, refit, and compare the held-out R-squared. If the number barely moves, the variable was earning its place. If it collapses, the variable was carrying the outcome, and the impressive fit was never about your marketing. A model that only holds up with branded search volume in it is reflecting demand that already existed, not the effect of your ads.
This has an uncomfortable consequence. The model without the leak usually shows a lower number than the one with it. You may end up choosing a model that reports 0.78 out of sample over a vendor's that reports 0.92, then defending the lower number to a finance team trained to read higher as better. The defense is the drop-and-refit test: show that the higher number collapses when the outcome-coupled variable comes out, and that the reallocations it implies would move budget the wrong way. A number you can defend beats a higher one that quietly misallocates spend.
From running this a lot, a strong out-of-sample R-squared for a marketing model, one that already beats a seasonal baseline, tends to land in the high 0.7s to high 0.8s depending on the business. Push much past 0.9 on a held-out window and the right move is to stop and audit the feature list, not celebrate. These are working bands, not laws of physics, and the number still has to defend itself.
What should you ask your measurement vendor?
Ask for both R-squared numbers, then ask how they stress-tested the model. One question does most of the work.
What was the out-of-sample R-squared, and how did you check for overfitting? If they only have an in-sample number, they graded their own answer key.
If they clear that, keep going. Does the model beat a seasonal-naive baseline, or is the R-squared just measuring the calendar? What is in the feature list, and what happens to the held-out fit when you pull branded search volume, site sessions, and affiliate spend out? And what was the out-of-sample MAPE, over which window? If the answer to any of these is a shrug, the model is not ready to move budget on.
A vendor who runs this on their own model and hands you the results is doing the job. A vendor who cannot produce an out-of-sample number is asking you to trust a fit you have no way to check.
What are the other checks?
The out-of-sample R-squared is the headline, but it does not stand alone. Two more numbers sit next to it.
One clarification, because it trips up people who also run geo tests: R-squared shows up in incrementality work too, but it is doing a different job there. In a geo test it measures how well your control markets tracked the treatment market before the intervention, a pre-period design check. In an MMM it measures out-of-sample prediction. It is the same statistic measuring two different things, so do not read across from one to the other.
| Check | What it tells you | Healthy | Red flag |
|---|---|---|---|
| Out-of-sample R-squared | Whether the model predicts data it never saw, vs a seasonal baseline | High 0.7s to high 0.8s, and beats seasonal-naive | Above ~0.9 on a fair holdout (audit the feature list) or barely beats the baseline |
| In-sample vs out-of-sample gap | How much fit is memorization | Small gap (e.g. 0.91 vs 0.89) | Large drop (e.g. 0.97 vs 0.62) |
| MAPE (held-out) | Average size of the model's miss, in percent | Roughly 10 to 15% | Over 20% |
| VIF | Whether channels are too tangled to trust the per-channel split | Under 5 | Above 5 on media variables |
MAPE, the average size of the model's miss in percent, is the plain-language read on prediction accuracy. As a working rule of thumb, not a cited standard, roughly 10 to 15% is strong and over 20% is a problem, on a held-out window. It has known quirks: it punishes over and under-forecasting unevenly and gets unstable when actuals run near zero, so it is a first screen, not the last word.
VIF, the variance inflation factor, tells you whether your channels are too tangled to trust the per-channel split. When Meta, Google, and promos all ramp together, the model can predict total revenue fine while getting the channel-level credit wrong. Above a VIF of 5, any precise per-channel ROI is precision the model has not earned. This is one of the ways a media mix model quietly fails even when the headline fit looks clean.
One detail ties back to leakage and belongs here: split the holdout chronologically, never randomly. Hold back the most recent stretch and predict forward. A random split lets the model train on weeks that sit right next to the ones it is being tested on, which is one of the easiest ways to let the answer leak in.
And do not stop at one holdout. A single held-out window can flatter or punish a model by luck of timing. Test across several rolling windows instead, each one training on earlier periods and predicting the next, then watch whether the error stays in a tight band or jumps around. A model that holds up across several of these is steadier than one tuned to a single window, and it forecasts more reliably going forward, which is the reason you built it.
What's your action plan?
Run a statistical check, then a business-logic check, before you move real money. The out-of-sample numbers tell you the model can predict. A clean holdout is not proof the model has the causal story right, so treat it as the gate to act, not the finish line.
- Statistical. Confirm the out-of-sample test was run, not just in-sample fit dressed up. First, check that the model beats a seasonal-naive baseline, otherwise the R-squared is measuring the calendar. Then interrogate any out-of-sample R-squared above 0.9 for leakage by auditing the feature list. Then check MAPE against the band and VIF on every media variable.
- Business logic. Get media buyers, finance, and analytics in one room. Black Friday should look like Black Friday in the decomposition. Saturation curves should bend. Baseline contribution should look sane for your category, often 25 to 70% of sales, though a brand with heavy organic or retail presence can legitimately run higher. If any of that is off, or the out-of-sample fit was unrealistically good, go back to the inputs before you trust a single number.
- Pilot. Start with 5 to 10% of budget, reversible. Track actuals against the prediction. Build the case study before the next quarterly review.
The difference between tools shows up here too. Some platforms are strong on continuous, always-on validation, and what separates them from a consultancy is the delivery model, not the math. A platform is something you run yourself. Stella is a measurement consultancy that builds and reviews every model with a data scientist, then hands you the same checks to grade our work with. Calibration against an experiment is a separate step from validation, not part of the out-of-sample check: one incrementality experiment before a major budget shift is the minimum bar. The rigor is the same, what differs is how much of it you run yourself.
Frequently asked questions
What is out-of-sample validation in an MMM?
It is holding back a stretch of recent history, refitting the model without it, then predicting that window and measuring the error. It checks whether the model generalizes to data it never saw, which in-sample fit cannot tell you.
Why is a high out-of-sample R-squared a bad sign?
Because overfitting drives out-of-sample R-squared down, not up, so a high held-out number cannot come from overfitting. It usually means one of two things: an easy test where seasonality does the work, or leakage, where a variable coupled to revenue in the same week let the model see part of the answer. Check that the model beats a seasonal baseline, then audit the feature list.
What causes leakage in a marketing mix model?
Outcome-coupled variables. Branded search volume, site sessions, and affiliate spend all track revenue mechanically rather than driving it. Included in a holdout, they turn a blind window into an open-book test.
What out-of-sample MAPE should I expect?
As a working rule of thumb, roughly 10 to 15% is strong and over 20% is a problem, measured on a held-out window. In-sample MAPE is not meaningful, since the model already trained on that data.
Do I still need experiments if the out-of-sample numbers look good?
Yes. Out-of-sample fit tells you the model predicts. It does not tell you the model has the causal story right. Calibration is a separate step, not part of out-of-sample validation: one experiment before a major budget shift is the minimum bar, and it is what separates a model you can act on from a hypothesis.