What is statistical significance in A/B testing?

A result is statistically significant when a difference that large would be unlikely if A and B really converted the same. At 95% confidence, that means a p-value below 0.05.

What p-value is significant?

One below the threshold you chose before the test: 0.05 for 95% confidence, 0.01 for 99%. A p-value of 0.036 is significant at 95% but not at 99%.

How many visitors do I need for an A/B test?

It depends on your baseline rate and the smallest lift you want to find. At a 4% baseline, detecting a 10% relative lift with 95% confidence and 80% power takes 39,475 visitors per variant.

How long should I run an A/B test?

Divide the total visitors needed by your daily visitors, then round up to whole weeks so every weekday counts the same. 78,950 visitors at 5,000 a day is 16 days, so run it for 3 weeks.

What is minimum detectable effect?

The smallest relative lift a test is sized to detect. Sample size grows with one over the effect squared, so halving the effect roughly quadruples the visitors you need.

Should I use a one-sided or two-sided test?

Two-sided, unless you decided beforehand that only an improvement matters. A one-sided test needs about 21% fewer visitors (31,095 against 39,475 per variant in the default example) but cannot show that B is worse.

What is sample ratio mismatch?

A traffic split that differs from the one you set more than chance allows, such as 10,600 against 9,400 on a 50/50 test. It signals a bug in assignment or tracking, so fix it before reading the result.

Free calculator

Statistical significance calculator for A/B tests

Check whether your A/B test result is real, see how sure you can be that B beats A, catch a broken traffic split, and plan how many visitors and days the next test needs.

Updated October 27, 2026

Your test

A (control)

Visitors

Conversions

B (variant)

Visitors

Conversions

Confidence level

Intended traffic to A

Test

Runs in your browser. Nothing you type is sent or stored.

Relative uplift of B

Significant

+15%

B converts at 4.6% against 4% for A.

p-value: 0.036
95% interval, B − A: +0.04 to +1.16 pts
Chance B beats A: 98.2%
Observed power: 55.2%
z-score: 2.09
Sample ratio: OK (p = 1)

Set the sample size before the test and read the result once, at the end. Checking repeatedly and stopping at the first significant result inflates false positives.

B beats A by 15%, and the difference is statistically significant at 95% (p = 0.036).

Quick answer

How do you know if an A/B test result is significant?

Run a two-proportion z-test. Pool the rate, p = (conversions A + B) ÷ (visitors A + B), compute z = (rate B − rate A) ÷ √(p(1 − p)(1/nA + 1/nB)) and turn z into a p-value. Below 0.05 is significant at 95% confidence. 400 conversions from 10,000 visitors against 460 from 10,000 gives z = 2.09 and p = 0.036: significant.

How to use the A/B test calculator

Plan the test
In Sample size, enter your baseline rate, the smallest lift worth finding and your daily traffic. The planner gives the visitors per variant and the days to run.
Run it to the end
Let the test reach that sample size before you read the result, without stopping early when it looks good.
Enter the results
In Test results, enter visitors and conversions for A and B, the confidence level and the split you intended.
Read and share
See the p-value, interval, chance to beat control and the sample ratio check. Copy the link to share the exact numbers.

How the calculator works

Test results run a two-proportion z-test: the difference in conversion rates divided by its standard error under the assumption that A and B convert the same (the pooled rate). The two-sided p-value asks whether B differs in either direction; the one-sided one only whether B is better. The confidence interval for B − A uses each arm’s own rate.

The chance that B beats A is Bayesian. With a flat Beta(1, 1) prior, each arm’s rate follows Beta(1 + conversions, 1 + non-conversions), and Evan Miller’s closed-form sum gives the probability exactly (above 20,000 conversions in both arms, a close normal approximation). Observed power is computed from the difference the test saw; Hoenig and Heisey (2001) show it is a one-to-one function of the p-value, so it adds no information of its own.

The sample ratio check is a chi-squared goodness-of-fit test of the visitor counts against the split you intended, the method Fabijan et al. describe. It flags p below 0.0005, the conservative threshold Microsoft’s Experimentation Platform uses.

The planner uses the standard sample size formula for two proportions (Fleiss). With more than one challenger, the Bonferroni correction divides the significance level by the number of comparisons with the control. As a cross-check, Kohavi et al.’s rule of thumb, n = 16σ² ÷ Δ² at 95% confidence and 80% power, gives 38,400 for the default example against the planner’s 39,475.

Formula

Rate = conversions ÷ visitors
Pooled p = (c_A + c_B) ÷ (n_A + n_B)
z = (p_B − p_A) ÷ √(p(1 − p)(1/n_A + 1/n_B))
CI = (p_B − p_A) ± z_(1−α/2) × √(p_A(1 − p_A)/n_A + p_B(1 − p_B)/n_B)
n per variant = (z_(1−α/2)√(2p̄(1 − p̄)) + z_(1−β)√(p₁(1 − p₁) + p₂(1 − p₂)))² ÷ (p₂ − p₁)²

Worked example

A: 400 conversions from 10,000 visitors = 4.00%. B: 460 from 10,000 = 4.60%.
Relative uplift: (4.60 − 4.00) ÷ 4.00 = +15%.
Pooled rate: 860 ÷ 20,000 = 4.3%. Standard error: √(0.043 × 0.957 × (1/10,000 + 1/10,000)) = 0.287 points.
z = 0.60 ÷ 0.287 = 2.09; two-sided p = 0.036, significant at 95%.
95% interval for B − A: +0.04 to +1.16 percentage points. Chance B beats A: 98.2%.

Why peeking breaks the test

Checking results as they come in and stopping at the first significant p-value inflates false positives. Evan Miller shows that peeking ten times turns what you think is 1% significance into 5%. Fix the sample size with the planner first, run the test to it, and read the result once.

Sample ratio mismatch

If you split traffic 50/50 but A got 10,600 visitors and B 9,400, something in assignment or tracking is broken, and the conversion numbers cannot be trusted either. A chi-squared test tells a real problem from noise: here p is far below 0.0005, while 10,120 against 9,880 (p = 0.09) is ordinary chance.

p-value or chance to beat control

The p-value answers one question: if A and B converted the same, how surprising is a gap this large? The Bayesian probability answers another: given the data and a flat prior, how likely is B’s true rate higher than A’s? The p-value comes with fixed error rates when you plan the test; the probability is easier to explain to a team.

A/B testing questions

What is statistical significance in A/B testing?: A result is statistically significant when a difference that large would be unlikely if A and B really converted the same. At 95% confidence, that means a p-value below 0.05.
What p-value is significant?: One below the threshold you chose before the test: 0.05 for 95% confidence, 0.01 for 99%. A p-value of 0.036 is significant at 95% but not at 99%.
How many visitors do I need for an A/B test?: It depends on your baseline rate and the smallest lift you want to find. At a 4% baseline, detecting a 10% relative lift with 95% confidence and 80% power takes 39,475 visitors per variant.
How long should I run an A/B test?: Divide the total visitors needed by your daily visitors, then round up to whole weeks so every weekday counts the same. 78,950 visitors at 5,000 a day is 16 days, so run it for 3 weeks.
What is minimum detectable effect?: The smallest relative lift a test is sized to detect. Sample size grows with one over the effect squared, so halving the effect roughly quadruples the visitors you need.
Should I use a one-sided or two-sided test?: Two-sided, unless you decided beforehand that only an improvement matters. A one-sided test needs about 21% fewer visitors (31,095 against 39,475 per variant in the default example) but cannot show that B is worse.
What is sample ratio mismatch?: A traffic split that differs from the one you set more than chance allows, such as 10,600 against 9,400 on a 50/50 test. It signals a bug in assignment or tracking, so fix it before reading the result.
What is observed power?: Power computed from the effect the test actually saw. Hoenig and Heisey (2001) show it is a one-to-one function of the p-value, so it adds nothing to it. Plan power before the test instead.

Cite this tool

Free to link and quote. Use this reference:

SaaSCity, "A/B Test Significance Calculator", October 27, 2026. https://saascity.io/free-tools/ab-test-calculator