Machine learning & feedback
Regression and overfitting: training is not testing
A curve can fit familiar observations extremely well and predict new ones poorly. Compare polynomial degrees on the same training and holdout data, and see when additional flexibility becomes harmful.
The short answer
Overfitting occurs when a model learns random details of its training sample. Training error then falls while error on independent data can rise. A fixed holdout reveals the difference, but repeated use for model selection turns it into validation data.
Try the free interactive calculatorFree · in your browser · no sign-up
At a glance
- Low training error does not establish good predictions for new data.
- Degree, noise and sample size jointly affect the fit.
- A holdout used to select a model must be followed by fresh test data for final evaluation.
Can a perfect fit predict badly?
A model must predict a target from an input . In this example, the underlying function is . Each observation also contains random noise. The application knows the true curve, but the fitted model receives only the training points. This lets us distinguish learning the pattern from explaining random deviations.
A straight line has little flexibility. A higher-degree polynomial can bend more and move closer to individual observations. The official scikit-learn demonstration also uses polynomial models to illustrate underfitting and overfitting (scikit-learn developers 2026). Our example uses its own synthetic data and a fixed, independently generated comparison sample. No observation comes from real customers or an uploaded dataset.
Keep training and holdout separate
The starting configuration contains 24 training observations and 80 holdout observations. Both samples follow the same underlying function and noise model, but use separate fixed random seeds. The calculator fits the curve exclusively to the training observations. Holdout target values do not enter the coefficient calculation.
The error measure is mean squared error: prediction minus observation is squared for each point, then the squared differences are averaged. Large deviations therefore receive more weight. Training error measures fit to data already used. Holdout error evaluates the same curve on different observations. Both include noise; even the true function would not exactly match every noisy target.
A simple noiseless check is . A first-degree regression can represent this relationship exactly and obtains zero error apart from numerical rounding. Our sine curve, however, is not a first-degree polynomial. A straight line systematically misses its waves even with many observations. That is underfitting caused by an unsuitable model form.
Experiment with degree, noise and sample size
Regression and overfitting – interactive calculator
Training observations determine the polynomial curve; 80 independently generated holdout observations evaluate it. Compare degrees one to twelve, noise levels and 16 to 60 training observations.
First increase degree without changing the data. The available polynomial spaces are nested: a higher-degree model can represent every curve available to a lower-degree model. Its minimum training error therefore cannot increase mathematically, although tiny numerical differences are possible. Holdout error has no such guarantee. A more flexible curve can swing sharply between observed locations.
Next increase noise. The same standardised random draws are simply scaled more strongly. This isolates the noise parameter instead of replacing the sample with a new random one. Increasing training size extends the existing training sample; the 80 holdout observations remain fixed. Compare the chart with the error table covering all twelve degrees.
Numerical computation is part of the model
High powers of similar values can make a naive coefficient calculation sensitive to rounding. The application therefore uses a Chebyshev basis on the interval from minus one to one and QR decomposition with reorthogonalisation. This leaves the space of representable polynomials unchanged while improving numerical computation. There is no hidden regularisation.
Numerical stability does not prevent statistical overfitting. A correctly calculated flexible curve can still predict poorly. The chart also does not artificially clip predictions to the height of the observed points: large excursions remain visible as part of the result. This experiment does not establish how the curve behaves usefully beyond the displayed input interval.
When the holdout stops being a final test
If you repeatedly try degrees and select the best holdout result, you use those points for model selection. They have effectively become validation data. An honest final performance assessment then requires fresh, untouched test observations or an appropriate nested evaluation procedure. The cross-validation documentation explains this separation and the risk of using test information through repeated selection (scikit-learn developers 2026).
Operational applications raise additional questions. Will later cases follow the same distribution? Were target values genuinely unknown at prediction time? Are there dependencies between dates or customers? A random split alone cannot answer those questions. Start with a simple benchmark, an appropriate separation in time and an error measure that reflects the business decision.
Record the evaluation design before comparing models. That makes it harder to quietly change the metric or sample until a favoured approach wins. Decision Intelligence connects model evaluation to the decision the model is intended to support, including the consequences of being wrong.
Frequently asked questions
Is a higher polynomial degree always worse?
No. An overly simple model can miss important structure. Extra flexibility may help first and later fit noise; suitable complexity depends on the data and task.
Why does changing sample size leave the holdout unchanged?
The 80 holdout observations use an independent fixed random seed. Adding training data extends only the training sample and preserves the comparison set.
Sources and further reading
- Onlinescikit-learn developers (2026): Underfitting vs. Overfitting.
- Onlinescikit-learn developers (2026): Cross-validation: evaluating estimator performance.
Content last reviewed on 14 November 2026. Please let us know about any errors: impressum@nyxai.com.