Sunflower Intelligence
Learn

Plate 19 of 19 · How I build the models

Validating a model: held-out data and cross-validation

About 6 minutes to read, then three questions that count toward your mastery.

A model always looks better on the data it learned from. Holding data back, and choosing the simplest model that is as good as the best, is how you find out what it will really do.

The idea

Error on the training data flatters a model, and a more complex model drives it ever lower. Error on held-out data, which the model never saw, falls at first and then rises once the model starts fitting noise. That rise is overfitting: the model has memorised the sample instead of the pattern.

One small held-out set is a shaky judge. k-fold cross-validation splits the data into k folds and holds each out in turn, so every row is tested once. Average the k errors, and measure their wobble with a standard error. The one-standard-error rule takes the simplest model whose average error is within one standard error of the best: models that close can’t really be told apart, and simpler ones travel better.

Every preparation step, such as scaling, filling gaps or choosing columns, must be fitted on the training part only. Otherwise information from the test rows leaks into training and flatters the score. For data over time, train on the past and test on the future.

The formula

SE = sfolds√k, choose the simplest model with CV ≤ CVbest + SEbest, E[training MSE] = σ2n − pn, E[new-data MSE] = σ2n + pn

sfolds, k
the standard deviation of the k fold errors, and the number of folds
CV
a model’s average error across the folds
n, p, σ
rows, fitted coefficients and noise, for a straight-line (least-squares) model; new data at the same inputs

A worked example

Plate 19

Five-fold cross-validation scores polynomial curves of degree 1 to 5 on the same data. Which degree should you choose?

  1. Average each degree’s five fold errors: 2.15, 1.24, 1.18, 1.17 and 1.28. The lowest is degree 4, at 1.17.
  2. Degree 4’s fold errors have a standard deviation of 0.212, so its standard error is 0.212 ÷ √5 = 0.095.
  3. The threshold is 1.17 + 0.095 = 1.265. Degrees 2, 3 and 4 all fall within it.
  4. Take the simplest of them: degree 2.

Degree 2’s error of 1.24 is within the noise of degree 4’s 1.17, and its curve bends far less, so it is the safer bet on new data. Degree 5’s rise to 1.28 is overfitting starting to show.

Try it

A least-squares model with p coefficients fitted to n rows, noise of 1. The gap is pure optimism: the model has fitted part of the noise.

40
5

How much training error flatters a model

1.13×new-data error ÷ training error
Expected training error (RMSE)
0.94
Expected error on new data
1.06
True noise
1.00

Plenty of rows for this many coefficients: training error is only mildly optimistic, though held-out data is still the honest test.

The common mistake

Letting the test data leak into training

Scaling the whole table before splitting lets the test rows set the means and standard deviations the model trains with, so the test score comes out too good. A stronger leak: a churn model that includes “refund issued”, a column recorded only after a customer cancels, can score near perfectly in testing and be useless in practice, because at prediction time that column is always empty. Fit every step on training rows, and use only what you would know at the moment of prediction.

Remember

  • Training error only falls with complexity; held-out error falls, then rises. The rise is overfitting.
  • Cross-validation tests every row once. Prefer the simplest model within one standard error of the best.
  • Fit scaling and every other preparation step on the training rows only, and never use information from after the prediction.

Check yourself

Three questions from the Learn room’s bank. Your answers update your mastery of this skill, the same as in Practise.

Use it on real numbers

Open Why I hold data back, in How I build the models and try the idea on your own numbers.