Interactive explainer

Bayesian A/B test calculator

Enter visitors and conversions to see how likely variant B is to be better, the expected cost of choosing wrongly and the plausible size of the uplift.

Interactive

Bayesian A/B test calculator

This calculator requires JavaScript. The model estimates each variant’s conversion rate with a Beta(1, 1) prior and binomial likelihood, producing P(pB>pA)P(p_B > p_A), expected loss and an uplift interval.

What the calculator computes

Each variant’s conversion-rate posterior is a beta distribution: with xx conversions among nn visitors and a flat prior, p∣x∼Beta⁡(1+x,  1+n−x)p \mid x \sim \Beta(1 + x,\; 1 + n - x). This yields three metrics:

  • P(pB>pA)P(p_B > p_A): the probability that B has the higher rate, calculated exactly with the closed-form formula for two beta distributions (Miller 2015).
  • Expected loss: the average conversion rate sacrificed when choosing one variant while the other is actually better. Choosing B gives E[max⁡(pA−pB,0)]\E[\max(p_A - p_B, 0)]. This supports a decision rule: choose a variant once its expected loss falls below a tolerance set in advance (Stucchio 2015).
  • Relative uplift pB/pA−1p_B / p_A - 1: a distribution from 40,000 Monte Carlo draws, with a median and 95% interval.

For comparison, the calculator shows the p-value from the classical two-sample z-test. It answers a different question: how surprising the data would be if both variants performed equally well.

Reading the result

  1. Set the rule first. Choose the expected-loss tolerance and minimum commercially relevant uplift before the test, not afterwards.
  2. Look beyond P(B>A)P(B > A). A high probability of a tiny advantage differs from a moderate probability of a large advantage. The uplift interval shows the scale.
  3. Check assumptions. The model assumes independent visits and a stable rate throughout the test. Unequal visitor allocation (sample ratio mismatch), novelty effects and multiple simultaneous metrics require separate consideration.

The Bayesian A/B testing article explains the derivation, common mistakes and when a test can be stopped.

Sources and further reading

  1. OnlineMiller, E. (2015): Formulas for Bayesian A/B Testing.
  2. ReportStucchio, C. (2015): Bayesian A/B Testing at VWO. Whitepaper, Visual Website Optimizer.

From NyxAI

smartlytics is the learning hub of NyxAI GmbH.

Explore the methods we work with, explained openly, with sources and calculations you can reproduce.

Decision Core

NyxAI is developing a Decision Core for Decision Intelligence that combines optimisation, simulation, statistics and reinforcement learning: compare options, make uncertainty visible and learn from outcomes, with people in control.

Use case: explain reporting variances with AI and evidence

DeepSquare

Explainable chess AI with human playing styles: NyxAI’s testing ground for understandable decisions under time pressure.