Interactive explainer

Interactive: Calibration: can predicted probabilities be trusted?

A well-calibrated forecast does not predict every event correctly. It reports reliable probabilities: among comparable cases assigned an 80% probability, the event should occur roughly eight times out of ten in the long run.

Interactive

Interactive: Calibration: can predicted probabilities be trusted?

Change sample size, overconfidence and correction temperature. Synthetic outcomes are generated locally using a fixed seed.

Reading the display

The dashed diagonal represents agreement between predictions and observed frequency. Points summarise ten fixed probability bins. The table lists case counts and both rates; empty bins are omitted. Small bins can deviate substantially from the diagonal even when the underlying probabilities are correct.

The Brier score measures mean squared error of individual predictions. ECE summarises grouped gaps. Neither is classification accuracy. Every number comes from this synthetic demonstration and does not evaluate a real product.

Three experiments

  1. Overconfidence: keep correction temperature at 1 and raise the overconfidence factor. Predictions become more extreme while referring to the same cases.
  2. The known correction: set both factors to 2. By construction, this restores the original model probabilities; observed points remain irregular because of chance.
  3. More cases: switch between 100 and 2,000 cases. Watch counts in the outer bins and examine how the curve becomes more stable or more variable.

Reset chooses 500 cases, factor 2 and temperature 1. In a real application, temperature must be estimated on separate data. The tool demonstrates that principle but does not train a model on your data. The calibration article explains practical evaluation and the limits of grouped metrics.

From NyxAI

smartlytics is the learning hub of NyxAI GmbH.

Explore the methods we work with, explained openly, with sources and calculations you can reproduce.

Decision Core

NyxAI is developing a Decision Core for Decision Intelligence that combines optimisation, simulation, statistics and reinforcement learning: compare options, make uncertainty visible and learn from outcomes, with people in control.

AI workflows at NyxAI

DeepSquare

Explainable chess AI with human playing styles: NyxAI’s testing ground for understandable decisions under time pressure.