Bayes & statistics

Bayesian A/B testing: comparing conversion rates

A Bayesian A/B test answers the questions that matter in a meeting: how likely is variant B to be better, by how much, and what does being wrong cost? A guide with a worked example, careful interpretation and a calculator.

  • 10 min read
  • Updated on
  • smartlytics editorial team

At a glance

  1. Each variant gets a beta posterior distribution for its conversion rate. This directly gives P(B > A), an uplift interval and the expected loss of a decision.
  2. A high probability that B is better says nothing by itself about the effect’s size. That requires an uplift interval and a relevance threshold chosen in advance.
  3. Bayes does not automatically protect against mistakes caused by repeated peeking. Choose the decision rule, minimum duration and sample-size plan before starting.

An A/B test is a randomised experiment: visitors are assigned at random to two variants, and a target metric, usually the conversion rate, is compared. The result supports a decision: adopt B or keep A. A classical significance test answers an indirect question: how surprising would this data be if there were no difference? A Bayesian A/B test answers the questions relevant to the decision: how likely is B to be better, by how much, and what risk am I taking?

This article builds on Bayesian statistics explained and works through a complete example. You can reproduce and change the same numbers in the Bayesian A/B test calculator.

The model: one beta distribution per variant

Each variant has an unknown conversion rate, pAp_A or pBp_B. Out of nn randomly assigned visitors, kk convert; the count follows a binomial distribution k∼Bin⁡(n,p)k \sim \Bin(n, p). With a beta distribution as prior, the posterior is another beta distribution:

p∣k∼Beta⁡(α+k,  β+n−k)p \mid k \sim \Beta(\alpha + k,\; \beta + n - k)

A useful starting point is the flat Beta⁡(1,1)\Beta(1, 1) prior for both variants. With thousands of visitors, its influence is negligible. Informative knowledge from earlier tests is allowed, but should be identical for A and B; otherwise the prior favours one variant before the first visitor arrives.

Variant A Variant B
Visitors 10,000 10,000
Conversions 420 468
Posterior Beta(421, 9,581) Beta(469, 9,533)
Posterior mean 4.21% 4.69%
95% credible interval 3.82%–4.61% 4.28%–5.11%

The two intervals overlap. That alone settles nothing: overlapping individual intervals do not constitute a test of the difference. The joint comparison of both posteriors is what matters.

Three metrics for a Bayesian A/B test

1. The probability that B is better

Because the two posteriors are independent, the probability that B has the higher rate is a simple integral over B’s density fBf_B and A’s cumulative distribution function FAF_A:

P(pB>pA)=∫01fB(x) FA(x)  dxP(p_B > p_A) = \int_0^1 f_B(x)\,F_A(x)\,\d x

It can be calculated numerically or, for integer parameters, exactly using Evan Miller’s formula (Miller 2015). Here B(⋅,⋅)\mathrm{B}(\cdot,\cdot) denotes the beta function:

P(pB>pA)=∑i=0αB−1B(αA+i, βA+βB)(βB+i) B(1+i, βB) B(αA, βA)P(p_B > p_A) = \sum_{i=0}^{\alpha_B - 1} \frac{\mathrm{B}(\alpha_A + i,\ \beta_A + \beta_B)}{(\beta_B + i)\,\mathrm{B}(1 + i,\ \beta_B)\,\mathrm{B}(\alpha_A,\ \beta_A)}

Both methods give 95.0% for the example. Given the model, prior and data, B has the higher conversion rate with 95% probability.

2. Expected loss

Probability alone ignores how much is at stake. Chris Stucchio therefore proposed expected loss as a decision metric (Stucchio 2015): how much conversion rate is lost on average if we choose B but A is actually better?

LB=E[max⁡(pA−pB,  0)],LA=E[max⁡(pB−pA,  0)]\mathcal{L}_B = \E\big[\max(p_A - p_B,\; 0)\big], \qquad \mathcal{L}_A = \E\big[\max(p_B - p_A,\; 0)\big]

In the example, choosing B has an expected loss of just 0.006 percentage points, about 0.14% of the baseline rate. Choosing A would have an expected loss of 0.49 percentage points. The decision rule is to choose a variant once its expected loss falls below a threshold set in advance at a level that is no longer material to the business. Stucchio calls this the “threshold of caring”.

3. Effect size

The decision also depends on how much better B is. Relative uplift u=pB/pA−1u = p_B / p_A - 1 has no simple closed-form distribution but is easy to simulate: draw many pairs from the two posteriors and calculate uu for each. With 400,000 draws, the example yields:

  • Median uplift: +11.4%
  • 95% interval: −2.0% to +26.7%
  • Probability of more than +5% uplift: 81.7%; more than +10%: 57.8%

This is the honest interpretation: B is probably better, but the effect’s size is still highly uncertain. Anything from slightly worse to substantially better remains plausible. Presenting “+11% uplift” alone leaves out half the information.

Interactive

Bayesian A/B test calculator

The calculator requires JavaScript. Example results: P(B > A) = 95.0%, expected loss when choosing B = 0.006 percentage points, uplift interval = −2.0% to +26.7%.

Bayesian and classical tests compared

The usual two-sample test for proportions gives z≈1,65z \approx 1{,}65 and a two-sided p-value of p≈0,099p \approx 0{,}099 for the same data. By the usual convention, this is “not significant at the 5% level”, yet B is better with 95% probability. Is that contradictory?

No. With a flat prior and large samples, P(pB>pA)P(p_B > p_A) is very close to one minus the one-sided p-value, which is 0.0497 here. A two-sided test at the 5% level therefore requires substantially more evidence than “a 95% probability that B is better”. Both methods see the same, fairly modest evidence; they express it differently. The p-value is not the probability that there is no difference (Wasserstein & Lazar 2016).

Classical test Bayesian A/B test
Question answered How surprising is the data if pA=pBp_A = p_B? How likely is B to be better, and by how much?
Example result p≈0,099p \approx 0{,}099 (two-sided) P(B>A)=95,0 %P(B > A) = 95{,}0\,\%
Effect size Confidence interval Credible interval for uplift
Decision risk Not given directly Expected loss
Planning Power analysis for a minimum effect size Simulation of the decision rule (pre-posterior)

Set decision rules in advance

A Bayesian A/B test produces probabilities; a decision needs a rule. Three common approaches are:

  1. A threshold for P(B > A): adopt B if P(pB>pA)≥0,95P(p_B > p_A) \ge 0{,}95. Easy to communicate, but it ignores effect size.
  2. Expected loss: adopt the variant whose expected loss falls below the “threshold of caring” (Stucchio 2015). This connects statistics directly to the cost of making the wrong decision, a central idea in decision theory.
  3. ROPE: define a region of practical equivalence, for example uplift between −1% and +1%. If the 95% highest density interval lies entirely outside it, the difference is considered relevant; if entirely inside, the variants are practically equivalent; otherwise the question remains open (Kruschke 2018).

Which rule you choose matters less than specifying it before the test, together with the metric, duration and threshold.

Peeking and stopping rules: what Bayes does and does not resolve

A common argument for Bayes is that you can check at any time. There is some truth in that. The posterior depends on observed data, not on the intention to stop, and is correctly calculated at every point (Rouder 2014). A sample size fixed in advance is therefore not required for a valid calculation.

A separate question is how often a decision rule is wrong across many experiments. We simulated 4,000 A/A tests, with no real difference and both rates at 4.2%. Results were checked every 1,000 visitors per variant, up to 20,000. A “winner” was declared as soon as P(pB>pA)P(p_B > p_A) exceeded 95% or fell below 5%:

  • With a single evaluation at the end: in about 11% of A/A tests (the expectation is 10%, 5% in each direction).
  • Stopping at the first favourable look across up to 20 evaluations: in about 42% of A/A tests.

The posterior is correct at each individual moment, but checking until a number looks attractive finds an effect far more often in experiments where none exists. How well optional stopping works depends on whether the prior matches the effects that actually occur and on precisely what is to be guaranteed (Deng et al. 2016; de Heide & Grünwald 2021). In practice, use a fixed duration, a small number of planned interim analyses, an expected-loss stopping rule with a minimum duration, or sequential methods designed for continuous monitoring (Johari et al. 2017).

How many visitors does a test need?

Bayesian tests also need planning. A classical approximation provides a useful starting point. Detecting a relative uplift of 10% at a baseline rate of 4.2%, meaning 4.20% versus 4.62%, with 80% power at the 5% two-sided level requires

n≈(z0,975+z0,8)2 [p1(1−p1)+p2(1−p2)](p2−p1)2≈37.500n \approx \frac{(z_{0{,}975} + z_{0{,}8})^2 \,\big[p_1(1-p_1) + p_2(1-p_2)\big]}{(p_2 - p_1)^2} \approx 37.500

visitors per variant. The practical online-experiment rule n≈16 σ2/δ2n \approx 16\,\sigma^2/\delta^2 gives almost the same answer, around 36,500 (Kohavi et al. 2020). For an uplift of just 5%, the requirement rises to about 147,000 per variant: half the effect, four times the sample.

Bayesian planning simulates the decision rule beforehand in a pre-posterior analysis: assume a plausible true effect, simulate many experiments and count how often the rule triggers. With a true uplift of 10% and 10,000 visitors per variant, only around 43% of 2,000 simulated experiments reach P(pB>pA)≥95 %P(p_B > p_A) \ge 95\,\%. The worked example was therefore small for an effect of that size, another reason to report the uplift interval.

Reporting the result

A good results record separates what the data shows from what was decided. For the example, it might read:

  1. Data and assumptions: 10,000 randomly assigned users per variant, flat Beta(1, 1) priors for both, allocation checked with no sample ratio mismatch, and a duration of two full weeks.
  2. Rates: A 4.21% (95% interval 3.82–4.61%), B 4.69% (4.28–5.11%).
  3. Comparison: 95% probability that B is better. Relative uplift: median +11%, 95% interval −2% to +27%.
  4. Decision: choosing B has an expected loss of 0.006 percentage points. If the threshold set beforehand was 0.01 percentage points, adopt B while explicitly noting that the effect’s size remains uncertain and will continue to be monitored in operation.

Everyone can then see how certain the result is, and the decision remains understandable even if the effect later proves smaller than hoped. This point is often underestimated: a good decision under uncertainty can have a disappointing outcome without having been a bad decision.

When Bayesian A/B testing is worthwhile

The Bayesian approach is particularly useful when a decision must be made and its risk needs to be tangible: probabilities and expected losses can be combined directly with costs, revenue or effort. Communication becomes easier because the results answer the question asked. It can also form part of larger models, such as hierarchical models across many tests or segments (Gelman et al. 2013).

When the aim is less to draw one clean conclusion and more to avoid losing revenue while learning, adaptive methods offer an alternative. The article on multi-armed bandits and Thompson sampling explains how they gradually direct traffic towards the better variant.

Conclusion

A Bayesian A/B test replaces “significant or not?” with three quantities suited to decisions: the probability that B is better, the expected loss of a choice and an interval for effect size. It does not justify arbitrary peeking or replace planning. It offers an honest, directly usable account of what the data actually shows.

Frequently asked questions

What is a Bayesian A/B test?

An A/B test that calculates a posterior distribution of the conversion rate for each variant, usually with a beta-binomial model. Instead of a p-value, it provides direct statements such as ‘B is better with 95% probability’, an uplift interval and the expected loss of choosing the wrong variant.

Does P(B > A) = 95% mean statistically significant?

No. That probability concerns the hypothesis given the model, prior and data; a p-value concerns the data under the null hypothesis. With a flat prior and large samples, P(B > A) is close to one minus the one-sided p-value. A two-sided test at the 5% level therefore requires more evidence than a 95% probability threshold.

Can I stop a Bayesian A/B test at any time?

The posterior is correctly calculated at every point. However, repeatedly checking and stopping at the first favourable result declares a winner much more often in experiments with no real difference. Use a duration chosen in advance, planned interim analyses or an expected-loss stopping rule with a minimum duration.

Which prior should I use for an A/B test?

Start with a flat Beta(1, 1) prior for both variants. With substantial traffic, its influence is negligible. Informative priors from previous tests are possible but should be the same for both variants so that prior knowledge does not favour one of them.

Sources and further reading

  1. OnlineMiller, E. (2015): Formulas for Bayesian A/B Testing. evanmiller.org.Derivation of the closed-form formula for P(B > A) in the beta-binomial model.
  2. ReportStucchio, C. (2015): Bayesian A/B Testing at VWO. Whitepaper, Visual Website Optimizer (VWO).Introduces expected loss and the threshold of caring as a decision rule.
  3. PaperKruschke, J. K. (2018): Rejecting or Accepting Parameter Values in Bayesian Estimation. Advances in Methods and Practices in Psychological Science 1(2), 270–280.
  4. BookKohavi, R., Tang, D. and Xu, Y. (2020): Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.The practical reference for online experiments, including sample-size planning and sample ratio mismatch.
  5. PaperRouder, J. N. (2014): Optional stopping: No problem for Bayesians. Psychonomic Bulletin & Review 21(2), 301–308.
  6. Paperde Heide, R. and Grünwald, P. D. (2021): Why optional stopping can be a problem for Bayesians. Psychonomic Bulletin & Review 28.
  7. PaperDeng, A., Lu, J. and Chen, S. (2016): Continuous Monitoring of A/B Tests without Pain: Optional Stopping in Bayesian Testing. IEEE International Conference on Data Science and Advanced Analytics (DSAA).
  8. PaperJohari, R., Koomen, P., Pekelis, L. and Walsh, D. (2017): Peeking at A/B Tests: Why it matters, and what to do about it. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD).
  9. BookGelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A. and Rubin, D. B. (2013): Bayesian Data Analysis (3rd edition). CRC Press.
  10. PaperWasserstein, R. L. and Lazar, N. A. (2016): The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician 70(2), 129–133.

Content last reviewed on 25 September 2026. Please let us know about any errors: impressum@nyxai.com.

From NyxAI

smartlytics is the learning hub of NyxAI GmbH.

Explore the methods we work with, explained openly, with sources and calculations you can reproduce.

Decision Core

NyxAI is developing a Decision Core for Decision Intelligence that combines optimisation, simulation, statistics and reinforcement learning: compare options, make uncertainty visible and learn from outcomes, with people in control.

DeepSquare

Explainable chess AI with human playing styles: NyxAI’s testing ground for understandable decisions under time pressure.