Curves, thresholds and restricted ranges

In depth

Reading the curve in bands

A quick way to see a curve’s shape without fitting anything is to group x into bands and average y within each band. The table does this for the iced share; with 612 café-days every band has at least 50 days, so each average is reasonably stable.

Read down the last column: small steps at the top of the table, large steps in the middle, small steps again at the bottom. A straight line through these averages would overstate the rise on cold days and on hot days and understate it in between. The departures from a fitted line, its residuals, would show the same pattern; B06 plots residuals properly.

Strength along the range: the threshold

Correlations within ranges confirm what the bands show: r = −0.10 for loads of 7 or less and 0.93 above 7. A hinge transformation makes the shape explicit: $h = \max(0, x - c)$ is zero below the threshold c and rises one for one above it. With c = 7, the hinge has r = 0.917 with the wait; squaring it, to allow for the wait rising ever faster, gives 0.980. That 0.98 is closer to the truth about this link than 0.83.

Two cautions. First, the threshold of 7 came from looking at these data, so a correlation with a hinge chosen to fit them flatters the fit; it describes this year’s log well and should be checked on new data. Second, Spearman’s ρ is 0.867 here, a little above r, because ρ rewards any steady rise, but it is no better at describing a flat stretch followed by a steep one. For a threshold, report where it is and how the slope changes, not only a coefficient.

U-shapes and turning points

For a symmetrical U, the positive products of z-scores on one side cancel the negative ones on the other, so r comes out near zero. In the hotel’s data, r is −0.97 on the cold side (settings up to 19.5°C) and +0.96 on the warm side; together, 0.00. Spearman’s ρ, which needs a consistent direction, fails too (−0.03). The fix is to measure against the shape: complaints have r = 0.97 with the distance from 19.5°C and 0.999 with its square. As with the threshold, the turning point was read from the data, so state it as a finding to confirm.

Transformations in outline

Some curves straighten under a transformation. A log transformation turns equal ratios into equal differences, which suits relationships in which y grows by a percentage when x grows by a percentage. In twelve fictional towns, the number of cafés rises with population but more and more slowly: r = 0.950 on the raw values, 0.994 on the logs of both. Logs need positive values, so they can’t be used directly on Lantern’s rainfall, which is zero on most days.

Spearman’s ρ is 0.993 for the towns, on raw values or logs alike, because a log keeps every value in the same order. That is a useful property: a rank correlation doesn’t depend on which monotonic transformation you choose, while Pearson’s r does. Fitting a curve after a transformation, and interpreting its slope, belongs to B06.

Restricted range

Range restriction means the data cover only part of the values the variable takes. If the relationship is a straight line with even scatter, and the restriction is on x alone, then shrinking the standard deviation of x from S to s changes r in a predictable way. With $k = S \div s$, the correlation over the full range would be $r_u = k r_r \div \sqrt{1 - r_r^2 + k^2 r_r^2}$, where $r_r$ is the correlation in the restricted data. The formula card gives the details.

For the summer days, k = 15.0 ÷ 6.4 ≈ 2.3 and the formula turns the summer r of 0.841 into 0.964, against 0.940 actually observed. It overshoots because the relationship is an S, not a line, and the scatter is not even. The formula is a guide to how much restriction matters, not a way to recover the “true” r.

Range restriction is everywhere in business data. Customers who were approved for credit, staff who were hired, products that stayed on the menu, stores that survived: each is a selected group with a narrower range than the population it came from, and correlations inside it are usually weaker than across everyone. The weaker correlation is accurate for the group; it is wrong only if read as a statement about the whole population.

Restriction does not always weaken r. If you select the part of a curve where it is steepest, r can rise: for loads above 7, r = 0.93, more than 0.83 for all days. And restricting to the steep part of a curve shrinks r far less than restricting to a flat part: days between 60 and 74°F still give 0.906 for the iced share, against 0.657 for July. The general rule is that r depends on the range you look at, so always say what range it describes.

Unreliable measurements attenuate r

Every measurement has some error. The door counter misses people in a crowd; a manager timing the queue with a stopwatch on five customers gets a noisy average. Random error adds scatter that has nothing to do with the relationship, so the observed correlation is weaker than the correlation between the true values. Spearman (1904) called this attenuation and gave a correction: if the reliability of each measure is known (the share of its variance that is true variation, often estimated by measuring the same thing twice and correlating the two sets of results), then $r_{\text{obs}} \approx r_{\text{true}}\sqrt{\text{rel}_x \, \text{rel}_y}$.

For example, with fictional figures: if the true correlation is 0.80, the load is measured with reliability 0.90 and the stopwatch waits with reliability 0.60, the observed r would be about 0.80 × √(0.90 × 0.60) = 0.59. Dividing an observed r by √(rel_x × rel_y) estimates the true one, but only as well as the reliabilities are known, and it can produce values above 1 when they are underestimated or when sampling error in r is large. Use the correction to judge whether a weak correlation might come from noisy measurement; better still, measure more reliably.