A/B Testing: Design, Sample Size and Analysis

An A/B test compares two versions of something, such as an old and a new checkout page, by randomly showing each version to a different group of users and measuring which performs better. Random assignment is what makes it powerful: if the groups differ only by the version they saw, a difference in results is caused by the change. A/B tests are the standard way product teams decide what to ship, and a favourite interview topic.

Design before you start

1. Hypothesis: “The new checkout increases purchase conversion.”
2. Primary metric: one number that decides the test, such as conversion rate per visitor.
3. Guardrail metrics: numbers that must not get worse, such as refunds or page load time.
4. Unit of randomisation: usually the user, so each person always sees the same version.
5. Sample size and duration, calculated in advance, in whole weeks to cover weekday and weekend behaviour.

Calculate the sample size

The sample size depends on the baseline rate, the smallest lift worth detecting, the significance level and the power. Smaller lifts need far more users.

Analyse the result

This example plans a test to detect a lift from 5.0% to 5.5% conversion, then analyses the results:

from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

# 1. Before the test: how many users per group to detect a lift from 5.0% to 5.5% conversion?
effect = proportion_effectsize(0.055, 0.050)
n = NormalIndPower().solve_power(effect_size=effect, alpha=0.05, power=0.8, alternative="two-sided")
print(f"users needed per group: {n:,.0f}")

# 2. After the test: results from the planned sample
conversions = [1_755, 1_602]          # new checkout (B), old checkout (A)
visitors    = [31_500, 31_500]
rate_b, rate_a = conversions[0] / visitors[0], conversions[1] / visitors[1]
z, p = proportions_ztest(conversions, visitors)
low, high = confint_proportions_2indep(conversions[0], visitors[0], conversions[1], visitors[1])
print(f"A: {rate_a:.2%}   B: {rate_b:.2%}   lift: {(rate_b - rate_a) * 100:+.2f} percentage points")
print(f"z = {z:.2f}, p-value = {p:.4f}")
print(f"95% confidence interval for the lift: {low * 100:+.2f} to {high * 100:+.2f} percentage points")

users needed per group: 31,217
A: 5.09%   B: 5.57%   lift: +0.49 percentage points
z = 2.71, p-value = 0.0066
95% confidence interval for the lift: +0.13 to +0.84 percentage points

How to read it:

• Detecting a half-point lift on a 5% baseline needs about 31,000 users per group, which is why small changes need large tests.
• The new checkout converted at 5.57% vs 5.09%, a lift of about half a percentage point (about 9% relative).
• The p-value of 0.0066 is well below 0.05, so the difference is unlikely to be chance.
• The confidence interval (+0.13 to +0.84 points) says the true lift is probably positive, but could be small. Whether that is worth shipping is a business decision, using the expected revenue and the cost of the change.

Check the test itself

• Sample ratio mismatch (SRM): if you planned a 50/50 split but got 52/48, something is broken in assignment or logging. Do not trust the result until it is explained.
• Balanced groups: check that key characteristics, such as device or country, are similar in both groups.
• Guardrails: confirm none got worse.

Common pitfalls

1. Peeking: checking results daily and stopping when they look significant greatly increases false positives. Stop at the planned sample size, or use a method designed for continuous monitoring.
2. Too many metrics or segments: some will look significant by chance. Decide the primary metric in advance and treat the rest as exploratory.
3. Too short: novelty effects and weekly patterns can mislead. Run whole weeks.
4. Wrong unit of analysis: randomising by user but analysing by session understates uncertainty.
5. Ignoring practical significance: a statistically significant tiny lift may not be worth the cost.

Key takeaway

Design A/B tests before starting: one primary metric, guardrails, randomisation by user, and a sample size calculated from the smallest lift worth detecting. Analyse with the right test, report the lift with a confidence interval, check for sample ratio mismatch, and avoid peeking. Practise explaining it with the data science interview questions.