Bayesian A/B: the probability that B beats A

A frequentist A/B answers evasively: "p = 0.04, we reject H0". The business wants it simple: "what is the probability that B is better than A, and how much do we lose if we're wrong?". The Bayesian approach answers these questions directly. Move the prior and the data — the posterior updates.

▏ posterior of A▏ posterior of BP(B > A) = 91.3%

Conversion A10%

Conversion B12%

n per group800

The two distributions are the plausible conversion values of A and B after the data. P(B>A) is the share of cases where a random value from B exceeds one from A. Larger n — both distributions get narrower and the answer more confident. A direct answer to "should we ship B", without a p-value.

The idea is the same as in the previous lesson, but for two variants. For conversion A and conversion B we build a posterior distribution each (from the prior + observed data). Now we have not two points, but two distributions of plausible conversion values.

you can flip beats with ← → keys

What it means

Bayesian A/B shines in business communication: "B is better with probability 96%, the expected loss of choosing B is 0.1%" is clear without statistical training and leads straight to a decision.

But the prior must be chosen honestly: a strong biased prior with little data will drag the conclusion. In practice weak (uninformative) priors are used so the data decides.

Where it shows up

Many A/B platforms offer a Bayesian mode precisely for the interpretable "probability to win" and "expected loss" — it reduces arguments about interpreting p-values.

A close relative is multi-armed bandits: there, the variants' posterior distributions are used to route more traffic to the better variant as the test runs.

Definitions

Per-variant posterior

the distribution of plausible metric values of a variant after the data (prior + likelihood).

P(B > A)

the probability that variant B is better than A; computed from the two posterior distributions.

Expected loss

the average loss from choosing a variant if it turns out worse; a stopping and decision criterion.

Bayesian vs frequentist A/B

the first gives "probability of being better" and "risk"; the second — Type I error control and the p-value.

When the method lies (assumptions)

"The probability that B is better" inherits the prior: with a weak prior on small data it easily shows 90%+ on pure noise. Fix the prior in advance rather than tuning it toward the desired result.

Bayesian A/B is gentler about peeking but not immune: constantly stopping at "95% better" still biases conclusions. And this probability says nothing about HOW MUCH better — look at the expected effect and the loss.