Outliers and influence

In depth

Why one point can move r so far

Pearson’s r is the average product of z-scores, $r = \frac{1}{n-1}\sum z_{x} z_{y}$, where each z-score is a value’s distance from its mean in standard deviations. Each point contributes its own term $z_{x} z_{y}/(n-1)$. You might expect the slip to damage r through a huge negative term, but its term is small and positive: its footfall is 18.3 standard deviations above the mean, its transactions only 0.71 above, and the product divided by 611 is 0.021.

The damage is done through the standard deviation. The slip raises the standard deviation of footfall from 87.1 to 129.9 people, so every other day’s z-score for footfall shrinks by about a third. In the corrected log all 612 terms add up to 0.981; with the slip, the other 611 add up to only 0.658. In short, r ≈ 0.981 × 87.1 ÷ 129.9 ≈ 0.66, plus the slip’s own 0.021. An outlier in x that does not follow the pattern lowers r mostly by inflating the spread, not by its own contribution.

The same reasoning explains the other two cases. A point far out on both variables and on the trend inflates both standard deviations but also adds a large positive product, enough to raise r. A point far out in y alone inflates the spread of y and adds little, so r falls.

Measuring influence: leave one out

An influential point is one whose removal would change a result noticeably. The simplest measure for a correlation is a leave-one-out analysis: compute r again without point i, call it $r_{(-i)}$, and look at the change $r_{(-i)} - r$ for every point. With 612 points that is 612 correlations, which software does in an instant (the engine behind this course’s widgets does exactly this).

The table shows how lopsided influence can be. Removing the slip changes r by +0.302; removing any other single day changes it by at most 0.007. When one point’s change dwarfs all the others, look at that point before anything else. Influence says how much a point matters, not whether it is wrong: the festival day of 29 May is the second most influential day here, and it is perfectly genuine.

Regression has fuller measures of influence, such as leverage and Cook’s distance, which look at how a point moves a fitted line; B06 covers them. For a correlation, leave-one-out changes are usually enough.

Finding outliers in two variables

Screening each variable on its own, with z-scores or the 1.5 × IQR fences from B02, catches points like the slip, which is extreme in footfall. It misses a point that is ordinary on each variable but unusual in combination: a day with 300 visitors and 30 transactions has an unremarkable footfall and an unremarkable number of transactions, yet a conversion rate no real day has. Use three checks together:

1. Plot. A scatterplot shows every kind of unusual point at once, including points that break the pattern without being extreme on either axis.
2. Screen each variable with z-scores or fences, remembering that these are conventions (|z| > 3 is common, not a law).
3. Check plausibility with a ratio or rule from the business. Transactions divided by footfall must lie between 0 and 1, and in practice between about a quarter and two thirds; iced plus hot drinks should be at least the transactions on most days; a wait can’t be negative. Ratios like these catch errors that no z-score will.

What to do, and what not to do

1. Check the source. Go back to the door counter’s export, the till report, the paper roster. Most extreme values in business data are either real events or recording slips, and the source usually says which.
2. Correct errors when the correct value is known, and record the change: what was wrong, what it became, and on what evidence.
3. When a value is impossible and the source is lost, set it to missing (CORREL skips text and empty cells, and the engine uses pairwise deletion) and say so. Don’t guess a replacement.
4. Keep genuine extremes, and report the correlation with and without them, with the reason the two differ.
5. Check with ranks. Spearman’s ρ uses only the order of the values, so a point can be at most the largest rank. For the raw log, ρ is 0.970 with the slip and 0.971 without; when Pearson’s r and Spearman’s ρ disagree sharply, a few points are usually responsible.

What not to do is delete points because the result looks better without them. A correlation computed after removing the points that “don’t fit” is a correlation of your expectations. Decide your rules for handling outliers before you see which way they push the result, apply them to every point, and report every exclusion.

Genuine extremes: the festival days

At Harbourside, footfall and the average wait at the morning peak have r = 0.673 across all 306 days. The five festival days had footfalls of 502 to 620, between 5.2 and 6.9 standard deviations above the café’s mean, when no other day passed 282. Their waits were long as well, from 4.7 to 15.3 minutes against a typical 2. They are far out on both variables and broadly on the trend, so they pull r up: without them r is 0.476, and leaving out 29 May alone takes it to 0.595.

The same five days pull a different correlation the other way. Footfall and the day’s high temperature have r = 0.436 at Harbourside with the festivals and 0.508 without: festival crowds were huge on warm, but not unusually warm, days, so the five points sit far above the temperature trend, outliers in y. One set of genuine days can raise one correlation and lower another, which is why the decision to keep or set them aside should be made for the question, and reported.