Transformations and influence

In depth

The log transformation in detail

Logs need y > 0; for counts that can be zero, B02’s $\ln(1 + y)$ is a common workaround, though its coefficients no longer read as percentages. Logging y changes three things at once: the form (effects multiply), the spread (errors proportional to the level become constant in logs) and the shape of the errors (a right skew in y often becomes symmetric in ln y). Check all three after the change; here the residual standard deviation in logs, 0.098, means a typical day is within about 10% of its predicted footfall, busy or quiet.

Logs can go on x instead, or on both. With $\ln x$ as the predictor, a 1% rise in x goes with a change of about b/100 in y. With logs on both sides, b is an elasticity: a 1% rise in x goes with about b% change in y. Choose the scale from how the effect works (fixed amounts or percentages) and from the residual plots, not from which version has the larger R².

Back-transforming a prediction

A log model predicts $\widehat{\ln y}$. Exponentiating gives $e^{\widehat{\ln y}}$, which estimates the median footfall for that day, not the mean, because the exponential stretches the upper half of the errors more than the lower. To estimate the mean, multiply by a correction: Duan’s smearing factor, the average of $e^{e_i}$ over the log model’s residuals, or $e^{s^2/2}$ if the errors are normal. For Harbourside both are 1.005, a 0.5% correction, small because s is small; with s = 0.5 it would be 13%.

Prediction intervals back-transform directly: exponentiate the two limits. The result is not symmetric about the prediction, and it widens with the level, which is the point. For a dry 75°F Saturday after the build-up the log model predicts a median of 227 people, with a 95% prediction interval from 187 to 276. For a dry 45°F Tuesday it predicts 103, from 85 to 124. The levels model gives each day the same ±33 people, too narrow for the Saturday and too wide for the Tuesday.

Polynomial and hinge terms

Lower AIC is better for models of the same y on the same rows. All four have three or fewer coefficients.

A polynomial is quick to try, but its shape is global: a squared term bends the whole curve, including where the data are flat, and higher powers swing wildly at the ends of the data. A hinge is local: it leaves the line alone below the threshold and changes it only above. The price is choosing the threshold. Ana’s came from B05’s bands, chosen before this fit; trying thresholds from 5 to 9 gives R² from 0.930 to 0.961, highest at 7, so the data agree. Choosing a threshold by searching many values and picking the best fit overstates how well the model will do on new data; if you search, say so, and check on data not used to choose it.

Interpret transformed terms through predictions, not single coefficients. In the squared-hinge model the effect of one more transaction per barista hour depends on where you start: below 7 it is the load slope, −0.005 minutes (indistinguishable from zero); at a load of 10 it is $-0.005 + 2 \times 0.226 \times 3 = 1.35$ minutes. The squared term on its own is not “the effect of load”.

Leverage

Leverage measures how far a row’s predictor values are from the centre of all the rows’. With one predictor, $h_i = 1/n + (x_i - \bar x)^2 / \sum (x - \bar x)^2$. Leverages lie between 1/n and 1 and average p/n, where p counts the coefficients; a common rule of thumb flags rows above 2p/n (here 2 × 2 ÷ 612 = 0.0065). For the slip, $h = 1/612 + (2566 - 186.8)^2 / 10{,}305{,}017 = 0.551$, 84 times the flag. With several predictors leverage comes from the hat matrix, but the idea is the same: an unusual combination of x values, such as a festival day on a weekend with heavy rain, can have high leverage even if no single value is extreme.

Studentized residuals

Residuals of high-leverage rows are small by construction, because the line is pulled towards them. Dividing by the standard deviation each residual should have puts them on one scale: the internally studentized residual is $r_i = e_i / (s\sqrt{1 - h_i})$. The externally studentized (deleted) version, $t_i$, uses s from the fit without row i, so a large error can’t hide by inflating s; it follows a t distribution on n − p − 1 df if the model is right. Penn State’s regression notes call a row with an externally studentized residual beyond ±3 an outlier; ±2 is a common, looser convention for a first look, especially in small samples. Treat both as warning flags, not rules.

Cook’s distance

Cook’s distance asks how much all the fitted values would move if row i were left out, scaled by the model’s precision: $D_i = \dfrac{r_i^2}{p} \cdot \dfrac{h_i}{1 - h_i}$. It is large only when both the residual and the leverage are. Rules of thumb vary: Penn State’s notes suggest a closer look above 0.5 and call values above 1 quite likely influential; others use 4/n. The most useful check is visual, a value that stands far apart from the rest.

The festival days show why both parts matter. In a log model of Harbourside’s footfall without a festival term, the five festival days have studentized residuals from 5.0 to 7.4, extreme outliers. But they are ordinary weekend days in their predictor values, with leverages from 0.015 to 0.024, and their Cook’s distances are 0.08 to 0.20: above the 4/n line (0.013), far below 0.5. They still matter: all five fell on weekends, and leaving them in without a term raises the weekend coefficient from 0.469 to 0.521 (60% to 68% more people). Once corrected, the most influential rows in the footfall-and-transactions regression are the same festival days (29 May has the largest Cook’s distance, 0.56): their footfall is high, and they sit below the pooled line because Harbourside converts about 47% of visitors where Millbrook converts 55%, a sign that the pooled model needs a café term.

What to do with an unusual point

1. Check it against the source. The door counter’s export showed 256 for 14 September: an error. Correct it, and record the change.
2. If it is real, ask whether the model is missing something. The festival days are real and different. A festival dummy gives them their own coefficient; the other coefficients are then almost exactly those from leaving the five days out (weekend 0.4691 either way), and the model still describes them.
3. Report with and without. When a real point moves a conclusion and no term explains it, show both fits and say which conclusions depend on it.
4. Never delete a point because it spoils the result, and don’t delete automatically by a cut-off. Rules such as |t| > 3 or D > 1 tell you where to look, not what to remove.