A/B Test Significance Calculator

Variant B beat variant A — but is it real or luck? This calculator runs the standard two-proportion z-test on your numbers, giving uplift, z-score and p-value, with a plain verdict at the 95% confidence level.

How to use this calculator

Enter visitors and conversions for each variant.
Press Test — you get both rates, the uplift, z-score and p-value.
Only call a winner when p < 0.05 and the planned sample size was reached.

Formula used

z = (p₂ − p₁) ÷ √( p̄(1−p̄)(1/n₁ + 1/n₂) ), p̄ = pooled rate

The two-proportion z-test asks: if both variants truly converted identically, how often would a gap this large appear by chance? That probability is the p-value; below 0.05 is the conventional 95%-confidence bar. Two-tailed, since B could have been worse.

Example calculation

Worked example

A: 150/5,000 (3.0%) vs B: 195/5,000 (3.9%) — a +30% relative uplift.

Pooled rate 3.45%, z = 2.47, p ≈ 0.013 → significant at 95% (not quite at 99%). Verdict: B genuinely outperforms, with ~1.3% probability this gap was luck.

What significance does and doesn't mean

A significant result means the observed gap would be rare if the variants were truly equal — it does not certify the uplift's size. Early stopping is the classic sin: checking daily and stopping at the first p < 0.05 can triple your false-positive rate, because random walks cross the line temporarily. Fix the sample size in advance (see the sample size calculator) and read the result once.

Mind practical significance too: with enough traffic, a +0.5% relative lift becomes statistically significant while being operationally irrelevant. And an insignificant result isn't proof of "no difference" — often it just means underpowered.

Why use this calculator?

Get the standard z-test verdict without a statistics package.
See uplift and p-value together — size and reliability are different questions.
Check both 95% and 99% thresholds for high-stakes decisions.

Frequently asked questions

What does p < 0.05 actually mean?

If both variants truly performed identically, a difference this large would occur less than 5% of the time by chance. It's evidence against 'no difference', not a 95% probability that B is better — a subtle but real distinction.

How long should I run an A/B test?

Until the pre-computed sample size is reached, and ideally over whole weeks (covering weekday/weekend cycles) — typically 2–4 weeks minimum. Duration by calendar alone without sample-size math is how underpowered tests happen.

My test isn't significant — is the variant useless?

Not necessarily: absence of significance isn't evidence of absence. Small true effects need large samples to detect. Either run longer per your sample-size plan, test a bolder change, or accept the difference is too small to matter.