Interactive explainer
Interactive: Calibration: can predicted probabilities be trusted?
A well-calibrated forecast does not predict every event correctly. It reports reliable probabilities: among comparable cases assigned an 80% probability, the event should occur roughly eight times out of ten in the long run.
Interactive: Calibration: can predicted probabilities be trusted?
Change sample size, overconfidence and correction temperature. Synthetic outcomes are generated locally using a fixed seed.
Reading the display
The dashed diagonal represents agreement between predictions and observed frequency. Points summarise ten fixed probability bins. The table lists case counts and both rates; empty bins are omitted. Small bins can deviate substantially from the diagonal even when the underlying probabilities are correct.
The Brier score measures mean squared error of individual predictions. ECE summarises grouped gaps. Neither is classification accuracy. Every number comes from this synthetic demonstration and does not evaluate a real product.
Three experiments
- Overconfidence: keep correction temperature at 1 and raise the overconfidence factor. Predictions become more extreme while referring to the same cases.
- The known correction: set both factors to 2. By construction, this restores the original model probabilities; observed points remain irregular because of chance.
- More cases: switch between 100 and 2,000 cases. Watch counts in the outer bins and examine how the curve becomes more stable or more variable.
Reset chooses 500 cases, factor 2 and temperature 1. In a real application, temperature must be estimated on separate data. The tool demonstrates that principle but does not train a model on your data. The calibration article explains practical evaluation and the limits of grouped metrics.