Statistical tests: which one to choose
Hypothesis testing is the general idea. But to turn "a difference in the data" into a number for "how random is this" (a p-value), you need a concrete tool — a statistical test. There are several, and the choice depends on what kind of metric it is and what its data looks like. Let's work through an analyst's three workhorse tests and the conditions for using them.
a continuous metric with symmetric spread — e.g. time on site
First pick the metric type — you will see what its data looks like in two groups. Then pick a test: the widget shows its reference (null) distribution and tells you whether it fits and why. The test is matched to the metric BEFORE the experiment.
First, an important thing that is often confused. An A/B test is a METHOD of running an experiment: split users fairly and compare. A statistical test (t-test, z-test, etc.) is the mathematical TOOL that, inside the A/B, computes whether the difference is significant. So the test is applied within the A/B, not instead of it — different levels, not synonyms. On the right is a picker trainer: we'll come back to it, but first let's cover the three tests.
Choosing a test is a decision about which assumptions you trust: if you doubt the shape of the data, take a nonparametric one and knowingly sacrifice power.
In practice, choosing a test is part of the test design. Conversion → the z-test (proportions). Average order value or time — the t-test, but carefully: these metrics are skewed, and with outliers you first handle the tail (winsorization, log, removing clear errors) and apply the t-test to the mean; Mann–Whitney is left for the case where a shift of the distribution matters rather than the mean. Choosing a test after looking at the data is a way to nudge the result, so it is fixed in advance.
All tests share a common framework: H0/H1, Type I and II errors, the p-value and a significance threshold. Only the statistic's formula and the applicability conditions change. Understand the framework and you easily master a new test (χ², analysis of variance and beyond).
Product analytics almost always reduces the choice to two branches: a share metric (conversion, a retention flag) → the z-test/χ²; a mean metric (revenue, time) → the t-test or nonparametrics under heavy tails.
Medicine and science choose a test by the same rules: normal measurements — the t-test, ordinal scales and small samples — rank tests. A wrongly chosen test gives a wrong p — and a wrong decision.
The t-test has a brewing pedigree: William Gosset invented it at the Guinness brewery to compare barley varieties on tiny samples. The company allowed publication only under a pseudonym — Student. Hence the "Student's t-test".
Definitions
t-test: needs approximate normality or a large n; sensitive to outliers and heavy tails. z-test for proportions: large samples (enough successes and failures in each group). Mann–Whitney: independent observations; compares a shift of distributions, not the means themselves.
Common to all: independence of observations and a test chosen in advance. Picking a test after looking at the data ("p is smaller with this one") inflates false positives.