Bayes & statistics
Calibration: can predicted probabilities be trusted?
A model reports a 90% chance of success, yet only six in ten cases succeed. A reliability diagram helps assess whether its probabilities provide a sound basis for planning and decisions.
The short answer
A well-calibrated forecast does not predict every event correctly. It reports reliable probabilities: among comparable cases assigned an 80% probability, the event should occur roughly eight times out of ten in the long run.
Explore the interactive exampleFree · in your browser · no sign-up
At a glance
- Accuracy and calibration answer different questions.
- A reliability diagram and Brier score reveal different aspects of forecast quality.
- Fit a correction on separate data, then assess it using observations not previously used for that purpose.
A system labels many cases “90% likely to succeed”. If only 60% subsequently succeed, that number is too optimistic for a resource decision. The system might still rank good and bad cases sensibly. Ranking and reliable probabilities are different properties. Calibration examines the second.
A statement about groups of forecasts
One outcome does not disprove a probability forecast. Even a success assigned 90% probability can occasionally fail. Assessment requires many comparable forecasts recorded before their outcomes. Stated probabilities can then be compared with observed frequencies.
Always predicting 50% can be calibrated for a process with an overall balanced outcome. It still distinguishes no easy cases from difficult ones. Good calibration alone therefore does not make a forecast useful. Conversely, a model with strong discrimination can be overconfident. Guo and colleagues’ investigation of modern classifiers highlights the practical relevance of this distinction (Guo 2017).
The reliability diagram
The interactive example divides forecasts into ten fixed probability bins. For each nonempty bin, the horizontal axis gives the mean prediction and the vertical axis the observed success rate. Agreement lies on the diagonal. A point at (0.8, 0.6) means an average prediction of 80% but an observed rate of 60%.
Binning is a visual summary. It can hide differences within a bin. With few cases, points fluctuate considerably; a jagged curve does not by itself prove that a model is poor. The table therefore reports the number of cases in every bin. Empty bins are omitted, not displayed as a zero success rate.
A score for individual forecasts
For binary outcomes , we use the Brier score
Lower values are better. Two certain, correct forecasts, 0 for a failure and 1 for a success, give zero. Reversing those certain forecasts gives one. Predicting 0.5 for both gives 0.25. These examples specify the binary scaling used here.
The score is not purely a calibration error. Sanders explains that probabilistic forecast quality contains several aspects (Sanders 1967). The tool also reports ECE: the absolute gaps between observed and predicted bin averages, weighted by bin size. Its magnitude depends on the chosen partition and should not be treated as a universal acceptance threshold.
Making overconfidence visible
Our synthetic world has known success probabilities between 5% and 95%. A binary outcome is drawn for each case. An overconfidence factor then stretches the prediction’s log-odds, turning moderate probabilities into more extreme ones. At the same sample size, the outcomes stay fixed so that comparisons concern the same cases.
A temperature corrects this stretching:
For , predictions become less extreme. In this constructed model, setting temperature equal to the overconfidence factor restores the original probabilities. The diagram still does not lie exactly on the diagonal because the observed outcomes contain sampling noise.
Interactive: Calibration: can predicted probabilities be trusted?
Compare forecasts with observed frequencies. A fixed seed, ten bins and fully synthetic cases make the effects of each control inspectable.
Moving from a demonstration to operational assessment
With real data, the correct temperature is unknown. It is fitted on a separate calibration dataset; final evaluation requires further data not used during fitting. Reusing the same data for adjustment and proof of improvement easily overstates the benefit. This example permits manual exploration of temperature and does not perform automatic model selection.
Forecast time, outcome definition and observation window must also match. Missing delayed outcomes can bias the evaluation before any scoring takes place. Examine relevant subgroups and changes over time. A correction from last year cannot guarantee reliable probabilities today.
Which decision should improve?
Reliable probabilities are especially useful when connected to costs. An information test or resource plan needs more than a ranking. Alongside scores and diagrams, record action thresholds and the consequences of false approvals. Uncertainty in a metric can also be examined with an appropriate bootstrap.
Compare models using the same held-out cases and the same outcome definition. A lower error on an easier population is not automatically an improvement. The aim is to support a concrete choice with probabilities whose practical meaning survives outside the training sample.
Frequently asked questions
Does high classification accuracy imply calibration?
No. The preferred class may often be correct while its assigned probability is too extreme. Guo and colleagues study this distinction and post-processing temperature corrections (Guo 2017).
Does the Brier score measure calibration alone?
No. Squared forecast error reflects more than agreement between predicted probabilities and observed frequencies. Useful evaluation also considers resolution and reference forecasts (Sanders 1967).
Sources and further reading
- PaperGuo, C., Pleiss, G., Sun, Y. and Weinberger, K. Q. (2017): On Calibration of Modern Neural Networks.
- PaperSanders, F. (1967): The Verification of Probability Forecasts.
Content last reviewed on 14 November 2026. Please let us know about any errors: impressum@nyxai.com.