Holdout test
An experiment where a randomly selected group is withheld from seeing your ads, and the two groups are compared.
Randomly split the eligible audience. One half can see the ads, the other cannot. Compare outcomes. The difference is what the advertising caused, with no attribution window required and no inference involved.
Why it matters for HVAC
It is the only method that answers the question every contractor actually has, and it is almost never run in this industry. The reason usually given is cost. The real obstacle is statistical power: a test needs enough people in each group to detect a difference, and Meta’s own conversion lift studies generally want around two hundred thousand users per cell.
A single-market contractor at five thousand a month cannot reach that, which is why this site scores agencies against the largest account scale they claim to serve rather than holding everyone to a test most of their clients could never fund.
How it is measured
Meta’s Experiments tool runs user-level conversion lift where volume allows. Below that scale, a geo holdout is the practical alternative. Either way the design has to be stated: what was randomised, for how long, and what outcome was measured.
A worked example
A portfolio operator running thirty locations has what a single contractor lacks: enough separate markets to randomise and enough volume to power the test. That is the one circumstance in this whole sector where a holdout is straightforwardly available, and the agency on this ledger describing exactly that portfolio published aggregate optimisation instead.
What it is not
It is not an A/B test between two ads. That compares two treatments. This compares treatment against nothing, which is a different and much harder question.
Where this is demonstrated
The Tracking and the Conversions API pillar scores every listed agency on what they document here, and gives you the questions that tell you which one you are being sold.