How it works
p̄ = (xA+xB)/(nA+nB). z = (pB−pA)/√[p̄(1−p̄)(1/nA+1/nB)]. Two-sided p-value = 2 × (1−Φ(|z|)).
ASO and experiments
A/B test significance calculator
Compare two proportions with a two-sided pooled z-test and inspect the difference, lift, and approximate p-value.
- Conversion A
- 5%
- Conversion B
- 6%
- Relative lift B versus A
- 20%
- Approximate two-sided p-value
- 0.001925
- Difference in percentage points
- 1
Statistical difference in this test; check direction and the planned horizon before deciding.
Requires at least 10 conversions and 10 non-conversions in each group. Repeated checking and stopping at significance increase false positives.
Worked example
500/10,000 versus 600/10,000 give 5% and 6%, 20% lift and an approximate p-value of 0.0019.
Assumptions and limits
Requires at least 10 conversions and 10 non-conversions in each group. Repeated checking and stopping at significance increase false positives.
Method reference: NIST: difference of proportions
Frequently asked questions
How do I use the a/b test significance calculator?
Compare two proportions with a two-sided pooled z-test and inspect the difference, lift, and approximate p-value.
Does significance guarantee B is better?
No. Check direction, effect size, randomization integrity, and the planned horizon. A p-value is not the probability that a variant is correct.
Do I need an account and are my inputs uploaded?
No account is required. Calculations run in your browser. Entered numbers and text are not included in tool analytics events; only the tool ID, language, and action are recorded.
Want to go past the numbers and see the real market?
Noctral shows estimated MRR, live ads, creators, and competitors so you can validate opportunities with real signals instead of projections.
See Noctral data
