TheCreativeTestThatWasn'taTest
Your agency says they're "testing creatives." What they mean is: they uploaded two ad variants, let them run simultaneously, and declared one the winner based on whichever got more clicks.
That's not a test. That's two things happening at the same time.
What a real test requires
- A hypothesis - what specifically are you testing and why?
- Isolation - one variable changed, everything else held constant
- Statistical significance - enough data to draw a real conclusion
- A meaningful metric - clicks are not the metric; profit per impression is closer
- A time horizon - long enough to account for day-of-week and audience variation
Why most tests fail
Most creative "tests" fail because they're not designed to learn. They're designed to produce a slide for the monthly report that says "we're testing." The test exists to demonstrate activity, not to generate insight.
Google's own ad rotation settings compound the problem. The algorithm has its own preference, and it will bias impressions toward whichever variant it thinks will perform - undermining your test before it starts.
The cost of bad testing
Every bad test produces a false conclusion. That false conclusion becomes the basis for the next creative brief. Which produces the next bad test. The cycle compounds - you're building your creative strategy on a foundation of noise, not signal.
If your agency can't tell you the statistical significance of their last creative test, they weren't testing. They were guessing.