Free tool · ASO and experiments

A/B test significance calculator

Compare two proportions with a two-sided pooled z-test and inspect the difference, lift, and approximate p-value.

Build your scenario

Runs in your browser. No signup.

Starting values are editable examples.

Conversion A
5%
Conversion B
6%
Relative lift B versus A
20%
Approximate two-sided p-value
0.001925
Difference in percentage points
1

Statistical difference in this test; check direction and the planned horizon before deciding.

Requires at least 10 conversions and 10 non-conversions in each group. Repeated checking and stopping at significance increase false positives.

Free, no sign-up. Your numbers stay in your browser.

How it works

p̄ = (xA+xB)/(nA+nB). z = (pB−pA)/√[p̄(1−p̄)(1/nA+1/nB)]. Two-sided p-value = 2 × (1−Φ(|z|)).

Worked example

500/10,000 versus 600/10,000 give 5% and 6%, 20% lift and an approximate p-value of 0.0019.

Assumptions and limits

Requires at least 10 conversions and 10 non-conversions in each group. Repeated checking and stopping at significance increase false positives.

Method reference: NIST: difference of proportions

See it in real data

ASO

Keywords and search rank in the app stores

See the feature

Frequently asked questions

How do I use the a/b test significance calculator?

Compare two proportions with a two-sided pooled z-test and inspect the difference, lift, and approximate p-value.

Does significance guarantee B is better?

No. Check direction, effect size, randomization integrity, and the planned horizon. A p-value is not the probability that a variant is correct.

Do I need an account and are my inputs uploaded?

No account is required. Calculations run in your browser. Entered numbers and text are not included in tool analytics events; only the tool ID, language, and action are recorded.

Want to go past the numbers and see the real market?

Noctral shows the estimated revenue, the ads and the paywalls of apps that already earn, so you decide with market data and not just projections.

See Noctral data