Free tool · no login
A/B test significance calculator for ads and landing pages
Enter visitors and conversions for two versions to see their conversion rates, the lift and whether the difference is statistically significant at the 95 percent level.
A/B test result
Enter your numbers to see the result.
Free tool by Telehealth Media · telehealthmedia.com
Planning a test? The sample size calculator tells you how many visitors you need first.
How it works
The calculator runs a two-sided, two-proportion z test: the standard test for comparing two conversion rates.
- Rate A = conversions A / visitors A; rate B the same for B
- Pooled rate = all conversions / all visitors
- Standard error = square root of pooled rate x (1 - pooled rate) x (1 / visitors A + 1 / visitors B)
- z = (rate B - rate A) / standard error
- p value = the chance of a difference at least this large if the two versions were really the same
Significant at the 95 percent level means the p value is below 0.05. It does not mean B is 95 percent likely to be better, and it does not say how big the real difference is.
Worked example
Version A had 5,000 visitors and 150 conversions; version B had 5,000 visitors and 190 conversions.
| Step | Result |
|---|---|
| Rate A | 3.00% |
| Rate B | 3.80% |
| Lift | 26.7% |
| z | 2.207 |
| p value | 0.0273 |
| Significant at 95% | Yes |
These numbers illustrate the method; enter your own.
Telehealth notes
- Comparing two ads inside one Meta ad set is not a randomized test: Meta sends each ad to the people it expects to respond. Use Meta's A/B test tool or a landing page split for a fair test.
- Decide the sample size before you start and do not stop the moment the result turns significant. Peeking inflates false positives.
- Test one change at a time on landing pages, such as the headline or the first intake question, so you know what caused the difference.
Questions
What p value is significant?
Below 0.05 is the usual threshold for 95 percent significance.
Can I A/B test ads by putting them in one ad set?
Not fairly. The platform targets each ad differently, so the groups are not random. Use the platform's A/B test feature instead.
How many visitors do I need?
Use the sample size calculator before the test. It depends on your baseline rate and the smallest lift you care about.
Are my numbers stored?
No. The calculation runs in your browser and nothing is sent to us.