Hypothesis Testing and P-Values Explained Simply

Two warehouses have different average delivery times in last month’s data. Is one really faster, or is the gap just random variation? Hypothesis testing answers that question in a standard way, and the p-value is its main output. It is one of the most tested topics in data science interviews, and one of the most misunderstood.

The steps

1. Null hypothesis (H0): there is no real difference. The warehouses have the same average delivery time.
2. Alternative hypothesis (H1): there is a difference.
3. Choose a significance level in advance, usually 5% (0.05).
4. Calculate a test statistic and a p-value from the data.
5. Decide: if the p-value is below the significance level, reject H0.

What a p-value means

The p-value is the probability of seeing a difference at least as large as the one observed, if the null hypothesis were true. A small p-value means the result would be surprising if there were no real difference.

It is not the probability that the null hypothesis is true, and it says nothing about how big or important the difference is.

A worked example

This example compares delivery times for two warehouses with a t-test, then answers the same question by brute force with a permutation test: shuffle the warehouse labels many times and count how often a random split gives a gap as large as the real one. That count is exactly what a p-value means:

import numpy as np
from scipy import stats

rng = np.random.default_rng(21)
# Delivery times (days) for two warehouses
a = np.round(rng.normal(3.2, 1.0, 60), 1)
b = np.round(rng.normal(3.7, 1.0, 60), 1)
print(f"warehouse A mean {a.mean():.2f}, warehouse B mean {b.mean():.2f}, difference {b.mean() - a.mean():.2f}")

t, p = stats.ttest_ind(b, a, equal_var=False)            # Welch's t-test
print(f"t-test: t = {t:.2f}, p-value = {p:.4f}")

# The same question answered by brute force: how often does random relabelling give a gap this big?
observed = b.mean() - a.mean()
pooled = np.concatenate([a, b])
gaps = []
for _ in range(10_000):
    rng.shuffle(pooled)
    gaps.append(pooled[60:].mean() - pooled[:60].mean())
p_perm = np.mean(np.abs(gaps) >= abs(observed))
print(f"permutation test: p-value = {p_perm:.4f}")

low, high = stats.ttest_ind(b, a, equal_var=False).confidence_interval(0.95)
print(f"95% confidence interval for the difference: {low:.2f} to {high:.2f} days")

warehouse A mean 3.25, warehouse B mean 3.67, difference 0.42
t-test: t = 2.45, p-value = 0.0156
permutation test: p-value = 0.0170
95% confidence interval for the difference: 0.08 to 0.76 days

How to read it:

• Warehouse B takes about 0.42 days longer on average.
• Both tests give a p-value of about 0.016: if there were no real difference, a gap this large would appear only about 1.6% of the time by chance. Below 0.05, so we reject the null hypothesis.
• The two methods agree closely, which shows the p-value is not magic: it is “how often would chance alone do this?”
• The confidence interval (0.08 to 0.76 days) shows the likely size of the difference. It does not include zero, which matches the significant result, and it shows the gap could be small or fairly large.

Which test to use

Situation	Test
Compare two group averages	t-test (Welch’s version, which does not assume equal variances)
Compare two proportions, such as conversion rates	Two-proportion z-test or chi-square test
Relationship between two categorical variables	Chi-square test
Very skewed data or small samples	Permutation test, or a non-parametric test such as Mann-Whitney U
More than two groups	ANOVA, then pairwise comparisons with a correction

Errors and power

• A Type I error (false positive) is rejecting H0 when there is no real effect. Its rate is the significance level.
• A Type II error (false negative) is missing a real effect. Power is the chance of detecting a real effect, and it grows with sample size.

Common mistakes

1. Reading the p-value as the probability the result is a fluke or the probability H1 is true.
2. Reporting only “significant” without the size of the effect and a confidence interval.
3. Testing many things and reporting only the significant ones (the multiple testing problem).
4. Treating “not significant” as “no difference”: the sample may simply be too small.

Key takeaway

A hypothesis test asks how surprising your data would be if there were no real effect; the p-value answers that. Choose the test for your data type, decide the significance level in advance, and always report the effect size and a confidence interval alongside the p-value. Next: putting it into practice with A/B testing.