A/B Testing
A/B Testing explained
Start with a specific question: does a clearer explanation help people choose an appropriate offer? Turn this into a hypothesis, variants and a suitable outcome measure. Additional quality guardrails can prevent more completed orders from being achieved at the expense of more incorrect orders.
Variants should run concurrently within the planned comparison. Define who is randomly assigned and how that assignment remains stable. Before interpreting results, check allocation and measurement. Random assignment does not automatically make participants representative of all future customers.
Plan the sample, meaningful effect size, analysis method and stopping rules. In a test with a fixed analysis point, stopping at the first favourable significance result biases the decision. Continuous decision-making requires an appropriate statistical method.
Consider effect size, uncertainty and practical importance together. Statistical significance is neither a guarantee of success nor a judgement of quality. If several elements change as a package, the test estimates the effect of that package; it does not automatically isolate each individual contribution.
Examples
Hypothetical application
A website tests two explanations for choosing a plan, with random assignment at account level. Alongside qualified purchases, the team tracks incorrect orders. If the new version produces more purchases but also more unsuitable plan selections, the team uses agreed quality guardrails to decide on the next revision.
Key Points
- Define the hypothesis, random assignment and analysis in advance.
- Read effect size together with uncertainty.
- Take quality guardrails and inconclusive findings seriously.
Practical application
Write a testable hypothesis and identify the decision that depends on it. Check technical assignment and data quality, document the analysis and limits, and retain negative and inconclusive findings too.
Useful measures
Estimated effect
Show the change in the predefined outcome with uncertainty.
Quality guardrails
Check errors, complaints or outcome quality, for example.
Experiment validity
Check assignment, measurement and deviations from the plan.
Common mistakes
- Declaring a winner at the first favourable interim result.
- Presenting subgroups selected afterwards as findings confirmed in advance.
- Mistaking measurement problems or a changing denominator for user effects.
Sources and context
- Microsoft Research: Pre-experiment stage
Experiment design, random assignment and quality checks.
- Microsoft Research: Post-experiment stage
Checking measurement data, trade-offs and interpretation limits.
- Fabijan et al.: Trustworthy analysis of online controlled experiments
Research on planning and analysis, including early stopping.
Frequently Asked Questions about A/B Testing
That depends on expected data volume, the meaningful effect size and the analysis method, among other factors. Plan duration and stopping rules for the question; a universal minimum number of days is insufficient.
No. A test can compare a package of changes. Its estimate then applies to that package. Separating individual contributions requires an appropriate design.
No. The result may be too uncertain to detect a meaningful difference. Consider the estimated effect and its uncertainty before claiming equivalence.
Loading related terms…
All TermsArticles about A/B Testing
Predictive Analytics in Marketing: Data-Driven Predictions for Better Decisions
How Machine Learning and Predictive Analytics are revolutionizing marketing decisions, from churn prediction and CLV models to demand forecasting.
Behavioral Data Fusion: How Merging Behavioral Data Is Revolutionizing Marketing
Behavioral Data Fusion connects data streams from web analytics, CRM, social media, and IoT into a holistic customer view. Learn why isolated data silos are marketing's biggest blind spot.
Dark Social: How to Measure and Leverage the Invisible 80% of Your Content Reach
Most of your content is shared through private channels, but invisible to analytics. Here's how to use Dark Social strategically.