Bayes & statistics
Bayesian statistics explained
How do you estimate a success rate from just a few observations? Bayesian statistics combines prior knowledge with new data and makes the remaining uncertainty visible. A business example, worked formulas and a free calculator show the path from prior to posterior.
The short answer
Bayesian statistics updates a probability distribution for an unknown quantity: the posterior is proportional to likelihood times prior. Alongside an estimate, it provides an interval and probabilities for decision thresholds, conditional on the chosen model assumptions.
Try the free Bayesian calculatorFree · in your browser · no sign-up
At a glance
- Bayes’ theorem combines existing knowledge (prior) and data (likelihood) into an updated assessment (posterior).
- With few observations, the whole distribution matters: an estimated success rate alone does not tell you how certain a decision is.
- A credible interval describes uncertainty about the parameter; it has a different meaning from a frequentist confidence interval.
Decisions almost never come with complete information. A conversion rate is only an estimate, a test result can be wrong, and a forecast has a spread. Bayesian statistics takes this uncertainty seriously: it represents unknown quantities as probability distributions and updates those distributions as data arrives. The approach traces back to Thomas Bayes, whose essay appeared posthumously in 1763 (Bayes 1763); Pierre-Simon Laplace generalised it and applied it widely shortly afterwards.
The beta-binomial example below provides the derivation. In the interactive Bayesian calculator, you can change the prior, observations and decision threshold.
Probability as a degree of justified belief
In the classical, frequentist interpretation, probability is a relative frequency across repeated, possibly hypothetical, trials: a fair coin lands heads in half its tosses in the long run. Unknown quantities, such as the true conversion rate of a landing page, are fixed numbers in this interpretation; only the data is random.
The Bayesian interpretation also allows probabilities for statements about unknown quantities: “There is a 90% probability that the conversion rate lies between 3% and 5%.” Probability becomes a measure of justified belief, governed by the rules of probability theory. Edwin T. Jaynes showed how these rules can be understood as a consistent extension of classical logic to uncertain statements (Jaynes 2003).
This does not make the answer arbitrary. Belief is tied to assumptions and evidence and must stand up to scrutiny against both.
Bayes’ theorem
The product rule immediately yields Bayes’ theorem for a hypothesis and observed data :
Its four components have established names:
- Prior : how plausible is the hypothesis before observing the data?
- Likelihood : how probable would this particular data be if the hypothesis were true?
- Evidence : the overall probability of the data across all hypotheses considered, acting as the normalising factor.
- Posterior : how plausible is the hypothesis after observing the data?
Because the evidence does not depend on , the theorem is often remembered as “posterior is proportional to likelihood times prior”:
An example almost everyone misjudges
A disease occurs in 1% of the people tested. A test detects 90% of those who have it (sensitivity) and gives a false alarm for 9% of healthy people (false-positive rate). A person tests positive. How likely are they actually to have the disease?
Many people, including professionals, intuitively estimate something near 90%. Gerd Gigerenzer and Ulrich Hoffrage showed that the correct answer is much easier to find using natural frequencies instead of percentages (Gigerenzer & Hoffrage 1995):
| Out of 1,000 people | Positive test | Negative test | Total |
|---|---|---|---|
| With the disease | 9 | 1 | 10 |
| Healthy | 89 | 901 | 990 |
| Total | 98 | 902 | 1,000 |
Only 9 of the 98 people testing positive have the disease: roughly 9%. Using Bayes’ theorem:
The posterior of 9.2% is much smaller than the sensitivity because the disease is rare: among the large healthy population, even a 9% false-alarm rate creates more positive results than among the small number of people with the disease.
The example also shows why Bayes is a learning procedure. If a second, independent test is also positive, the previous posterior becomes the new prior:
After two positive results, the probability is roughly fifty-fifty. This calculation assumes that the two tests’ errors are independent conditional on disease status. Repeating the same test with the same source of error does not satisfy that assumption.
From hypotheses to parameters
In practice, the question is rarely about just two hypotheses. It concerns an unknown number: a success rate, an average delivery time or an effect. For a continuous parameter , Bayes’ theorem uses densities instead of probabilities:
The structure is the same. Again, the denominator only normalises the result, ensuring that the area under the posterior equals 1.
The prior: making assumptions visible
The prior describes which values are plausible before seeing the data. It can be flat (all values equally plausible), weakly informative (excluding implausible values without imposing much information), or informative (summarising reliable knowledge from earlier studies). A prior is not a trick for wishing a result into existence: it is a testable assumption. It can be justified, documented and tested through prior predictive checks, simulating the data it would imply before looking at real observations (Gelman et al. 2020).
The likelihood: what the data says
The likelihood is the same function used by frequentist methods. With the data held fixed, it describes how well each possible parameter value explains the observations. For successes in independent trials with success probability , this is the binomial distribution:
As a function of , the likelihood is not a probability density: its area need not equal 1. Multiplying by the prior and normalising turns it into a distribution.
The posterior: prior times likelihood
The calculation is especially clear with a beta distribution as the prior. Its density is proportional to . Multiplying it by the binomial likelihood simply adds the exponents:
The posterior is therefore another beta distribution:
Prior and posterior belong to the same family, so the beta prior is called conjugate to the binomial likelihood. The parameters can be interpreted as pseudo-observations: “successes” and “failures” representing information available before the new data.
The posterior mean is a weighted average of the prior mean and the observed rate:
Here : the data counts twice as much as prior knowledge, and the estimate lands between 20% and 35%. As grows, approaches zero and the data dominates. This pull towards prior knowledge is called shrinkage and protects against extreme estimates, particularly in small samples.
Live Bayesian update: prior, likelihood and posterior
The interactive explainer shows the prior, normalised likelihood and posterior of a success rate. Without JavaScript, the calculation above still applies: prior Beta(2, 8), 7 successes in 20 trials, posterior Beta(9, 21).
What the posterior provides
As a complete distribution, the posterior directly answers many questions:
- Point estimates: mean, median or mode; the mode is also called the MAP estimate. The best choice depends on which errors are costly, a question of decision theory.
- Intervals: a 95% credible interval contains with 95% probability, given the model, prior and data. Common choices are the central interval (2.5th and 97.5th percentiles) and the highest density interval (HDI).
- Probabilities of statements: is simply the posterior’s area to the right of 0.25.
- Predictions: the posterior predictive distribution averages over all plausible parameter values. In the beta-binomial model, the probability that the next trial succeeds equals the posterior mean. A flat prior gives Laplace’s rule of succession .
These direct statements explain why Bayesian methods fit decisions so well. “How likely is variant B to outperform A?” becomes a straightforward integration. The Bayesian A/B testing article shows the practical application.
Bayesian vs frequentist: a fair comparison
Both schools use the same probability calculus and often the same models, but they answer different questions:
| Frequentist | Bayesian | |
|---|---|---|
| What is uncertain? | The data, with a fixed unknown parameter | The parameter, represented by a distribution |
| Typical output | Estimate, confidence interval, p-value | Posterior, credible interval, probabilities of statements |
| Meaning of a 95% interval | The procedure covers the true value in 95% of repetitions | The parameter lies in the interval with 95% probability, given the model and prior |
| Existing knowledge | Indirectly through design and model choice | Explicitly through the prior |
| Strength | Controlled error rates over many repetitions | Direct statements for a particular decision |
| Typical risk | Misinterpreting p-values as probabilities of the null hypothesis | Unjustified or unchecked priors |
A p-value is the probability, under the null hypothesis, of observing a result at least as extreme as the actual result. It is explicitly not the probability that the null hypothesis is true. The American Statistical Association warned against this and other misinterpretations in a widely cited 2016 statement (Wasserstein & Lazar 2016).
In practice, the numbers are often closer than the philosophical differences suggest: with ample data and a weak prior, credible and confidence intervals are frequently almost identical numerically. The difference then lies mainly in interpretation and in the statement a decision actually requires.
When no closed-form solution exists
Conjugate pairs such as beta and binomial are convenient but are more the exception than the rule. For realistic models with many parameters, the posterior usually cannot be expressed in a simple formula. Instead, it is simulated: Markov chain Monte Carlo methods generate a sequence of parameter values whose frequencies represent the posterior. The idea dates back to the Metropolis algorithm (Metropolis et al. 1953); modern tools such as Stan use Hamiltonian Monte Carlo with the No-U-Turn Sampler (Hoffman & Gelman 2014). The Monte Carlo simulation article explains how random samples produce reliable numerical results.
Common mistakes and how to avoid them
- Hiding the prior. Every prior should be documented and justified. A sensitivity analysis shows whether the result depends on that choice.
- Forgetting the model. A posterior is valid only within its model. Posterior predictive checks compare simulated and real data to reveal misspecification (Gelman et al. 2013).
- Ignoring the base rate. The diagnostic-test example demonstrates how strongly the starting probability shapes the answer.
- Reporting single numbers instead of distributions. A mean without an interval hides precisely the information that Bayesian analysis is meant to preserve.
- Assuming Bayes resolves every stopping-rule question. The posterior remains correctly calculated during ongoing evaluation. The long-run error rates of a decision rule across many experiments are a separate question that should be addressed in advance.
Conclusion
Bayesian statistics rests on one rule: posterior is proportional to likelihood times prior. It leads to a way of working that exposes existing knowledge, carries uncertainty as a distribution and produces statements directly suited to decisions. For further study, McElreath offers an accessible introduction (McElreath 2020), Kruschke provides many worked examples (Kruschke 2015), and Gelman and colleagues supply the comprehensive reference (Gelman et al. 2013).
Frequently asked questions
What is the difference between a prior and a posterior?
The prior describes assumptions about an unknown value before observing data. The posterior is the distribution after observing data: multiply the prior by the likelihood of the observed data, then normalise. Today’s posterior can become tomorrow’s prior.
Is Bayesian statistics subjective?
It makes assumptions explicit instead of hiding them. Every analysis involves assumptions about the model, data selection and stopping rule; in a Bayesian approach, some appear explicitly in the prior. Good practice means justifying priors, checking them with prior predictive checks and testing how sensitive results are to them.
What is a credible interval?
A 95% credible interval contains the unknown parameter with 95% probability, given the model, prior and data. This is precisely the statement often incorrectly attributed to a frequentist confidence interval.
When is a Bayesian approach useful?
Especially when prior knowledge is available, data is scarce, a decision needs direct probability statements such as ‘How likely is B to outperform A?’, or uncertainty needs to feed into simulations and optimisation.
Sources and further reading
- PaperBayes, T. (1763): An Essay towards Solving a Problem in the Doctrine of Chances. Philosophical Transactions of the Royal Society of London 53, 370–418.
- BookJaynes, E. T. (2003): Probability Theory: The Logic of Science. Cambridge University Press.Probability as extended logic: the philosophical and mathematical foundation of Bayesian reasoning.
- BookGelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A. and Rubin, D. B. (2013): Bayesian Data Analysis (3rd edition). CRC Press.The standard reference; the authors provide a PDF edition for non-commercial use.
- BookMcElreath, R. (2020): Statistical Rethinking: A Bayesian Course with Examples in R and Stan (2nd edition). CRC Press.An exceptionally clear introduction focused on model building.
- BookKruschke, J. K. (2015): Doing Bayesian Data Analysis (2nd edition). Academic Press.
- PaperGigerenzer, G. and Hoffrage, U. (1995): How to improve Bayesian reasoning without instruction: Frequency formats. Psychological Review 102(4), 684–704.
- PaperWasserstein, R. L. and Lazar, N. A. (2016): The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician 70(2), 129–133.
- PaperMetropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H. and Teller, E. (1953): Equation of State Calculations by Fast Computing Machines. The Journal of Chemical Physics 21(6), 1087–1092.
- PaperHoffman, M. D. and Gelman, A. (2014): The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research 15, 1593–1623.
- PaperGelman, A., Vehtari, A., Simpson, D. et al. (2020): Bayesian Workflow. arXiv:2011.01808.
Content last reviewed on 14 November 2026. Please let us know about any errors: impressum@nyxai.com.