Holdout Tests: The Only Way to Prove Google Ads Actually Works
Attribution models tell you where conversions came from. Holdout tests tell you which conversions would have happened anyway. One is storytelling. The other is science.
The Uncomfortable Truth
Most Google Ads "success" is measured by Google's own attribution. It is like asking your advertising agency if advertising works. The incentives are misaligned, and everyone knows it except the people writing the cheques.
What Attribution Actually Measures
Attribution models answer a simple question: which touchpoints preceded a conversion? Last-click gives all credit to the final interaction. Data-driven distributes credit based on historical patterns. Multi-touch spreads it across the journey.
None of them answer the question that actually matters: would that customer have bought anyway?
A customer searching for "Nike Air Max size 10 UK" has already decided to buy. They have chosen the product, the size, and quite possibly the retailer. Your branded search ad might capture the click, but you are paying for a sale that was already yours.
"Attribution tells you who touched the ball before the goal. Incrementality tells you whether the goal would have been scored without that player on the pitch."
The Holdout Test Methodology
A holdout test is conceptually simple: split your audience into two groups. Show ads to one group. Withhold ads from the other. Measure the difference in conversion rates.
The gap between the two groups represents true incrementality. The conversions that happened only because of advertising. Everything else was cannibalisation, brand halo, or coincidence.
Control Group
No ads shown. Natural conversion rate establishes baseline. This group represents what would happen if you stopped advertising entirely.
Test Group
Ads shown as normal. Conversion rate measured against control. The difference is your true advertising lift.
Why Agencies Avoid Holdout Tests
Holdout tests require sacrificing some revenue to measure accurately. For the duration of the test, you are deliberately not showing ads to potential customers. This feels wrong, and it conflicts with quarterly targets.
More importantly, holdout tests often reveal uncomfortable truths. That expensive brand campaign? Might be 60% cannibalising organic traffic. Those retargeting ads with amazing ROAS? Might be claiming credit for sales that would have happened during the abandoned cart email sequence.
Agencies paid on percentage of spend have zero incentive to prove that less spend would work better. The misalignment is structural.
Practical Implementation
Google offers Conversion Lift studies through their platform. These use randomised user assignment and measure lift at a statistically significant scale. The methodology is sound, but the minimum spend requirements exclude smaller advertisers.
For brands without access to Google's native tools, geographic holdouts offer an alternative. Turn off advertising in one region while maintaining spend in similar regions. Compare revenue per capita. The precision is lower, but the insight is still valuable.
Holdout Test Checklist
- •Minimum 2-week test duration for statistical significance
- •Matched control and test populations by demographic and behaviour
- •No other marketing variables changing during test period
- •Measure both conversion rate and revenue per user
- •Account for delayed conversions beyond test window
What Good Incrementality Looks Like
Healthy incrementality varies by campaign type. Brand campaigns should show 20-40% true lift. Generic campaigns should exceed 60%. Prospecting should approach 80-90%.
If your brand campaign shows 15% incrementality, you are spending 85 pence to claim credit for something you already owned. That is not marketing. It is expensive accounting.
The CFO Conversation
When finance asks "how do we know Google Ads is working?", attribution data is not the answer. Attribution proves correlation. Holdout tests prove causation.
Running regular holdout tests transforms marketing from a cost centre that claims credit into a profit centre that proves value. The discipline is uncomfortable, but the credibility it creates with the board is invaluable.
The Bottom Line
If your agency has never proposed a holdout test, ask why. If they resist the idea, ask what they are afraid of finding. The answer tells you everything about whether they are optimising for your profit or their fee.
Want honest measurement?
We build incrementality testing into every engagement.
Get our insights in your inbox
Plain-English thinking about Google Ads. No spam, unsubscribe anytime.