A/B test significance calculator
Find out if version B of your ad, email or page really performs better than version A, or if the difference could be chance. Enter the visitors and conversions of each version and choose a confidence level.
Help: how to use it and FAQ
How to use it
- You enter the data
- Smallo works it out
- You get the result
- Enter the visitors and conversions for version A and version B.
- Choose the confidence level, usually 95%.
- Read the conversion rates, the lift and whether the result is significant.
Frequently asked questions
What does statistically significant mean?
It means the difference is unlikely to be chance. At 95% confidence, a difference this large would show up less than 5% of the time if both versions really performed the same.
Which test does the calculator use?
A two-sided two-proportion z-test, the standard test for comparing conversion rates. It compares the gap between the rates with the gap you would expect from random variation.
What if the result is not significant?
Keep the test running until each version has enough visitors and conversions, or accept that the difference is too small to matter. Decide the sample size before you start, and do not stop the test the moment it looks good.