Multiple regression

In depth

The model

The population model behind a multiple regression is $y = \beta_0 + \beta_1x_1 + \dots + \beta_kx_k + \varepsilon$, with errors $\varepsilon$ that average zero at every combination of the x’s. The fitted coefficients $b_0, \dots, b_k$ estimate the $\beta$’s. There are $p = k + 1$ coefficients counting the intercept, and the residual standard error is $s = \sqrt{\text{SSE}/(n - p)}$, in y’s units (the standard error of the regression). The conditions for the t tests and intervals are those of the inference-for-coefficients lesson, applied to the model as a whole; the assumptions-and-diagnostics lesson checks them.

The model is additive: each predictor’s slope is the same whatever the values of the others. On a weekend and on a weekday the high temperature is assumed to add the same 1.66 people per °F. When that is false (the cafés respond differently to the weekend, for example) the model needs an interaction term, which the next lesson adds.

A partial slope is a slope of residuals

There is a concrete way to see “holding the others fixed”. Regress y on all the other predictors and keep the residuals; regress $x_1$ on the same predictors and keep its residuals. The least-squares slope of the first set of residuals on the second is exactly $b_1$ from the multiple regression (a result known as the Frisch–Waugh–Lovell theorem). The residuals of $x_1$ are the part of $x_1$ the other predictors can’t predict, so $b_1$ is estimated only from that part. B05’s partial correlation is the correlation of the same two sets of residuals; the partial slope is their slope, in y’s units per unit of $x_1$.

This explains a practical rule. A slope can be estimated well only if its predictor varies once the others are held fixed. If a predictor is almost a copy of the others, its residuals are tiny and its slope is poorly determined: that is multicollinearity, the subject of the multicollinearity-and-model-choice lesson.

Omitted-variable bias

Suppose the true model contains $x_1$ and $x_2$, and you fit $x_1$ alone. The short slope is $\tilde b_1 = b_1 + b_2\delta$, where $\delta$ is the slope of $x_2$ on $x_1$. Two conditions make the bias: $x_2$ must matter for y ($b_2 \neq 0$) and it must go with $x_1$ ($\delta \neq 0$). Its direction is the sign of $b_2 \times \delta$: festivals add footfall and go with warm days, so leaving them out pushes the temperature slope up. Leaving out something unrelated to the included predictors, like the weekend column for the temperature slope, costs precision (a larger s) but causes no bias.

Not every extra predictor reduces bias. Adding a variable that lies on the path from x to y (a mediator, in B05’s terms) removes part of the effect you want; adding one caused by both x and y (a collider) creates a false link. The choice of predictors is a judgement about how the business works, which the multicollinearity-and-model-choice lesson takes further.

A trend left out

New cafés take time to find their customers. The generator planted a build-up: each café’s trade starts at 72% of its eventual level on 1 March and reaches 100% after four months. The weather follows the season, and March and April were both cold and quiet; so a model without the build-up credits some of the build-up to temperature. The third model’s residuals show it: they average −21.5 people in March and −13.8 in April, and +9.5 to +11.4 from October to December. The model over-predicts the early months and under-predicts the late ones.

The fourth model adds a build-up column that rises evenly from 0 on 1 March to 1 on 29 June and stays at 1, the shape the generator used. Its slope, 35.5 people, is the build-up’s size; the temperature slope falls to 1.44, and the monthly residual averages shrink to within ±4 people. The planted effect is multiplicative, so it differs from day to day: from about 0.4 people per °F on a wet weekday in March to about 5 on a festival day. Fitting the fourth model to the generator’s planted mean footfall for each day gives a temperature slope of 1.47 people per °F, the average effect the linear slope estimates; the fourth model recovers it and the third overstates it by about 13%. With real data no one tells you the trend’s shape. A straight-line “day” column is the wrong shape here: R² rises, but the temperature slope moves further from the truth, to 1.79; plotting residuals in time order, as the assumptions-and-diagnostics lesson does, is how you find out what is missing.

R² and adjusted R²

R² keeps its meaning, the share of the variation in y around its mean that the fitted values account for: $R^2 = 1 - \text{SSE}/\text{SST}$. It can never fall when a predictor is added, because least squares can always set the new slope to zero and do no worse. So R² rewards adding anything, even a column of random numbers. Adjusted R-squared charges for each coefficient:

For the third model, $1 - (1 - 0.9221) \times 305/301 = 0.9211$. Adjusted R² rises when a new predictor’s t statistic is larger than 1 in size and falls otherwise, so it can go down. It is a rough guard against padding a model, not a test, and it is not a share of anything: it can even be negative. The next lesson but one compares models more carefully.

The overall F test

Each slope’s t test asks whether that predictor adds anything, holding the others fixed. The overall F test asks whether the predictors together add anything at all: $H_0: \beta_1 = \beta_2 = \dots = \beta_k = 0$. Its statistic compares the variation explained per predictor with the residual variation per degree of freedom:

on k and n − p degrees of freedom. For the third model, $F = (0.9221/4) \div (0.0779/301) = 891$ on 4 and 301 df, and the p-value (Significance F in Excel) is far below 0.001. With one predictor F is the slope’s t squared; with several it is a single test of all of them. The two kinds of test can disagree: when two predictors carry the same information, F can be large while neither t is, because the data show that something matters but can’t say which. The general F test that compares any two nested models is in the multicollinearity-and-model-choice lesson.

Reading the output in Excel

The ToolPak’s Regression tool takes an Input X Range of several adjacent columns and prints the same three blocks as for one predictor: Regression Statistics (Multiple R, R Square, Adjusted R Square, Standard Error, Observations), the ANOVA table (Regression, Residual and Total rows with df, SS, MS, F and Significance F) and one row of coefficients per predictor with its standard error, t Stat, P-value and 95% limits. The Explore widget prints it for any set of Harbourside’s predictors.

LINEST accepts several x columns too, but returns the coefficients in reverse order: for x’s in columns C, D and E, =LINEST(B2:B31,C2:E31) spills $b_E, b_D, b_C, b_0$ across one row (Microsoft’s documentation states the order as m_n, m_n−1, …, m₁, b). To pick out the slope on column C, take the third entry: =INDEX(LINEST(B2:B31,C2:E31),1,3). With stats set to TRUE, row 2 holds the standard errors in the same order, row 3 R² and s, row 4 F and the residual df, and row 5 SSR and SSE. TREND with several x columns predicts from the fitted equation: =TREND(B2:B31,C2:E31,G2:I2) for a new day whose x values are in G2:I2.

Standardised slopes, with care

Slopes come in their predictors’ units (people per °F, per inch, per festival), so their sizes can’t be compared directly. A standardised slope rescales each to standard deviations, $b_j^{*} = b_j\,s_{x_j}/s_y$: the change in y, in SDs of y, per SD of $x_j$. For the fourth model they are 0.31 for the high, −0.21 for rain, 0.45 for the weekend, 0.61 for the festival and 0.16 for the build-up.

They are easy to over-read. A 0/1 column’s SD depends on how common the 1s are, not on any “typical change”: the festival column’s SD is only 0.13 because festivals are rare, and its 0.61 means 330 people times that small SD. Rain’s SD is inflated by a few wet days, since most days are dry. And standardising uses this sample’s spreads, so the same effect gets a different standardised slope in a sample with a narrower range. Use them to describe one sample, not to rank causes.