Multicollinearity and model choice

In depth

The variance inflation factor

Regress predictor $x_j$ on all the other predictors and call that regression’s R² $R_j^2$. The variance inflation factor is $\text{VIF}_j = 1/(1 - R_j^2)$. It is the factor by which collinearity multiplies the variance of $b_j$ compared with a predictor unrelated to the others, so the standard error is multiplied by $\sqrt{\text{VIF}_j}$. With two predictors $R_j^2 = r^2$: for Harbourside, $r^2 = 0.97064$ and $1/(1 - 0.97064) = 34.1$, and $\sqrt{34.1} = 5.8$.

The standard error of a slope is $\text{SE}(b_j) = \dfrac{s}{\sqrt{\sum (x_j - \bar x_j)^2}}\sqrt{\text{VIF}_j}$. Comparing transactions alone with transactions and footfall, the residual standard error s barely changes (28.1 to 27.6), so the standard error grows by about √VIF: 0.275 ÷ 0.048 is about 5.7, which is exactly $\sqrt{34.1} \times 27.6/28.1$. A VIF of 34 means that the 306 days give as much information about transactions’ separate slope as about 9 days would if the two predictors were unrelated.

Penn State’s STAT 462 notes treat VIFs above 4 as worth investigating and above 10 as signs of serious multicollinearity; these are conventions, and other sources use 5 and 10. A pairwise correlation can miss collinearity among three or more predictors (one can be close to a combination of two others), which the VIF catches. Collinearity is also built into models with interactions: a dummy and its product with a numeric predictor are highly correlated, which inflates the main effect’s standard error; centring the numeric predictor reduces it without changing the fit.

What to do about it

• Ask which question you are answering. Footfall, transactions and sales are a chain: people come in, some buy, purchases make sales. For “what is a visitor worth?”, regress sales on footfall alone ($3.18 per person); for “what is a transaction worth?”, on transactions alone ($6.64). With both in, footfall’s slope asks something narrower: at a given number of transactions, does a busier café sell more? Its $0.44 per person is consistent with the generator’s planted rise in the average ticket on busy days (about $0.22 per visitor at Harbourside’s typical trade, once the generator’s ticket rise is put in the log’s scaled dollars), but the interval is wide.
• Don’t drop a predictor only because its slope isn’t significant. If it is a confounder, dropping it brings back omitted-variable bias. Collinearity makes slopes imprecise, not wrong; a wide interval is the honest answer.
• Combine or get better data. Two measures of the same thing can be combined into one (a ratio such as transactions per visitor, or an average). More rows, or rows where the predictors move apart, are the only real cure.
• For prediction, it hardly matters. The model with both predictors predicts Harbourside’s sales as well as either alone, as long as the two keep moving together as they did in the data.

Comparing nested models: the general F test

Two models are nested when one (the reduced model) is the other (the full model) with some terms removed. The nested F test asks whether the removed terms together improve the fit by more than chance: $H_0$: all their coefficients are zero. With SSE from each fit, q terms removed and the full model’s residual degrees of freedom,

on q and $n - p_\text{full}$ df (the general linear F test of Penn State’s notes). The overall F test is the special case in which the reduced model has no predictors; with one term removed, F is that term’s t squared. The test matters most for groups of terms, such as a set of dummies, which no single t test can judge. All the comparisons below use the same 612 café-days.

AIC and BIC are from the Gaussian log-likelihood, as statsmodels and the B06 engine report them. Lower is better. Only differences between models fitted to the same rows mean anything.

B against A adds one term: $F = 27.8$ on 1 and 604 df, which is the interaction’s t squared ($(-5.27)^2$); p below 0.001. C against B adds the build-up: $F = 330$, p below 0.001. D against C replaces the weekend dummy and its interaction with six weekday dummies and six interactions, ten extra coefficients: $F = 0.70$ on 10 and 593 df, p = 0.73. The ten extra terms add nothing beyond what the weekend dummy already says, which the generator planted (weekdays alike, weekends alike). The rows share their days, as before, so all these p-values are somewhat too small.

AIC and BIC

Information criteria compare models that need not be nested, as long as they are fitted to the same rows and the same y. For least squares, apart from a constant that is the same for every model on the same data,

The first term rewards fit and the second charges for each coefficient; the model with the lowest value is preferred. BIC’s charge, $\ln n$ per coefficient (6.4 for 612 rows), is stronger than AIC’s 2 whenever n is 8 or more, so BIC favours smaller models (Taboga’s StatLect notes describe AIC as the mildest penalty and BIC as the strongest). In the table both pick model C. A difference of a few units is not decisive; a convention many analysts use treats differences under about 2 as negligible. Neither criterion says whether the best model is good, only which of the candidates is least bad by its rule.

Overfitting and the hold-out check

A model with enough terms can follow the noise in its own data, fitting the rows it was estimated from very well and new rows badly. That is overfitting, and R² on the fitting data can’t detect it. The direct check is a hold-out: fit the model on one part of the data (the training set) and measure its errors on another part it never saw (the test set), for example by the root mean squared error, $\text{RMSE} = \sqrt{\tfrac{1}{m}\sum (y - \hat y)^2}$ over the m test rows.

Ana fits three models of Harbourside’s footfall on March to September (214 days) and tests them on October to December (92 days). A is the multiple-regression lesson’s third model (high, rain, weekend, festival); B adds the build-up; C keeps the high, rain, festival and the build-up and throws in everything else that could be tried: weekday dummies, squares of the high and rain, their interactions, and a cubic trend in days since opening, 17 predictors.

Test RMSE in people. AIC and BIC from the training fit.

Model C has the best training R² and the lowest AIC, yet its test RMSE is 78.6 people, nearly five times B’s 16.5. Its cubic trend in the day, fitted to the spring build-up and the summer, bends sharply down after September: on the test days it under-predicts by 66 people on average. Model A also under-predicts, by 24 people on average, for a different reason: without the build-up its temperature slope is too steep, so it puts the colder autumn days too low. B, which BIC prefers, has a test RMSE close to its training s (16.5 against 16.7), though the comparison flatters it a little (see the caution below).

Explaining or predicting

The two aims choose predictors differently. Peak barista hours are rostered in advance from the manager’s expectation of the day’s trade, so they are known before the day and help forecast footfall: adding them to model C raises adjusted R² from 0.9401 to 0.9420. But the roster carries the calendar and the weather with it, so the weekend’s coefficient falls from 68.8 to 58.1 and the festival’s from 330.7 to 290.6. Nobody would say the roster brings people in. For a forecast, the roster is a useful summary of what the manager knows; for explaining what the weekend or a festival is worth, holding the roster fixed takes away part of the answer.

For explanation, choose predictors from how the business works, before looking at p-values: confounders in, mediators and colliders out (B05’s rules), with a stated reason for each. For prediction, any predictor available at the time of the forecast is fair, but the model must be judged on data it hasn’t seen, and a coefficient in it shouldn’t be read as an effect.

Why choosing predictors by p-value misleads

Stepwise selection adds or removes predictors one at a time by their p-values (or AIC) until no change passes the rule. It is automatic and popular, and its results mislead. Ana tries 31 meaningless candidates: a dummy for each day of the month, added one at a time to Harbourside’s model B. Two pass the 5% test (the 27th and the 29th of the month), about the 1.6 that chance alone predicts (31 × 0.05). A stepwise procedure would keep them and report their p-values as if they had been the only ones tried.

Selection also picks proxies. Harbourside’s loyalty-app membership, a running total that grows through the year, is “highly significant” added to model A (t = 10.4), because it tracks the build-up. With the build-up in the model, its p-value is 0.55. Frank Harrell’s often-quoted list of stepwise’s problems (reproduced in a Stata FAQ) includes R² values biased high, p-values without their proper meaning, confidence intervals that are too narrow, slopes biased away from zero and severe problems with collinear predictors. The cure is the one above: decide the candidate predictors from the question, test a few planned comparisons, and check any model chosen from the data on data that played no part in choosing it.