Davies Meyer – home
    Analytics2 min read

    Holdout Testing

    Holdout testing excludes a defined group from a specific activity. Comparison with a treated group helps estimate the activity’s additional impact. Other activities can continue unchanged in both groups.

    Holdout Testing explained

    A newsletter generates many orders. How many would have happened without sending it? A holdout provides a planned comparison condition.

    **Define groups in advance.** Assign participants randomly where possible. Do not retrospectively compare readers with non-readers: they may already differ in purchase intent. Analysis should account for planned assignment and actual implementation.

    **Plan size and duration for an informative test.** A small fixed holdout percentage is not a reliable rule. Baseline rate, expected effect, variation, purchase cycle and acceptable opportunity cost determine the design. Define the time window and relevant quality outcomes in advance.

    **Check contamination.** Excluded participants may receive the same message through other channels. Document overlap and changes. The finding describes the tested difference within the existing marketing mix, not automatically the impact of all advertising.

    Examples

    Hypothetical application

    A shop randomly assigns eligible customers. One group receives the planned newsletter; the other does not receive this send. Orders and returns are measured over the same window. Other regular communication remains comparable.

    Key Points

    • Exclude a specific activity.
    • Randomise assignment where possible.
    • Justify group size through study planning.
    • Document overlap and implementation.

    Practical application

    Choose a clearly separable activity and check exclusion logic, group assignment and outcome data before .

    Useful measures

    Absolute lift

    Estimated additional impact against the holdout, adjusted for comparable group sizes.

    Implementation fidelity

    Check whether the planned inclusions and exclusions were actually followed.

    Common mistakes

    • Defining readers and non-readers retrospectively as experimental groups.
    • Choosing a small control without statistical power planning.

    Sources and context

    Frequently Asked Questions about Holdout Testing

    A randomised comparison with an untreated control condition can be designed as an A/B test. A/B tests are not limited to two creative variants.

    Related links

    Performance Marketing AgencyMarketing automation & CRMDigital Agency

    Loading related terms…

    All Terms

    Articles about Holdout Testing

    Predictive Analytics in Marketing: Data-Driven Predictions for Better Decisions
    Data & Analytics13 min

    Predictive Analytics in Marketing: Data-Driven Predictions for Better Decisions

    How Machine Learning and Predictive Analytics are revolutionizing marketing decisions, from churn prediction and CLV models to demand forecasting.

    February 4, 2026Read
    Behavioral Data Fusion: How Merging Behavioral Data Is Revolutionizing Marketing
    Data & Analytics12 min

    Behavioral Data Fusion: How Merging Behavioral Data Is Revolutionizing Marketing

    Behavioral Data Fusion connects data streams from web analytics, CRM, social media, and IoT into a holistic customer view. Learn why isolated data silos are marketing's biggest blind spot.

    January 15, 2026Read
    Dark Social: How to Measure and Leverage the Invisible 80% of Your Content Reach
    Data & Analytics11 min

    Dark Social: How to Measure and Leverage the Invisible 80% of Your Content Reach

    Most of your content is shared through private channels, but invisible to analytics. Here's how to use Dark Social strategically.

    February 10, 2026Read