Interactive explainer
Bayesian A/B test calculator
Enter visitors and conversions to see how likely variant B is to be better, the expected cost of choosing wrongly and the plausible size of the uplift.
Bayesian A/B test calculator
This calculator requires JavaScript. The model estimates each variant’s conversion rate with a Beta(1, 1) prior and binomial likelihood, producing , expected loss and an uplift interval.
What the calculator computes
Each variant’s conversion-rate posterior is a beta distribution: with conversions among visitors and a flat prior, . This yields three metrics:
- : the probability that B has the higher rate, calculated exactly with the closed-form formula for two beta distributions (Miller 2015).
- Expected loss: the average conversion rate sacrificed when choosing one variant while the other is actually better. Choosing B gives . This supports a decision rule: choose a variant once its expected loss falls below a tolerance set in advance (Stucchio 2015).
- Relative uplift : a distribution from 40,000 Monte Carlo draws, with a median and 95% interval.
For comparison, the calculator shows the p-value from the classical two-sample z-test. It answers a different question: how surprising the data would be if both variants performed equally well.
Reading the result
- Set the rule first. Choose the expected-loss tolerance and minimum commercially relevant uplift before the test, not afterwards.
- Look beyond . A high probability of a tiny advantage differs from a moderate probability of a large advantage. The uplift interval shows the scale.
- Check assumptions. The model assumes independent visits and a stable rate throughout the test. Unequal visitor allocation (sample ratio mismatch), novelty effects and multiple simultaneous metrics require separate consideration.
The Bayesian A/B testing article explains the derivation, common mistakes and when a test can be stopped.
Sources and further reading
- OnlineMiller, E. (2015): Formulas for Bayesian A/B Testing.
- ReportStucchio, C. (2015): Bayesian A/B Testing at VWO. Whitepaper, Visual Website Optimizer.