Correlation matrices

In depth

What a correlation matrix is

For m variables, the correlation matrix is the m × m table whose cell in row i and column j is $r_{ij}$, the correlation of variable i with variable j. It has 1s on the diagonal, $r_{ij} = r_{ji}$, and $m(m-1)/2$ distinct off-diagonal cells. A Spearman matrix holds Spearman’s ρ in each cell instead; the rank-correlation lesson explains when to prefer it. For the log the two tell the same story: high temperature and iced share are 0.94 by Pearson and 0.96 by Spearman, because the S-shaped curve is monotonic but not straight.

Not every symmetric table of numbers between −1 and 1 can be a correlation matrix. If x and y correlate at 0.9 and y and z at 0.9, then x and z must correlate at least $0.9 \times 0.9 - \sqrt{(1 - 0.9^2)(1 - 0.9^2)} = 0.62$: variables that both follow y closely must follow each other. The general bound for any three variables is in the Formulas section. A valid matrix is what mathematicians call positive semidefinite; you rarely need the term, but you will meet its failure when a matrix is assembled from different sources or different rows.

Pairwise deletion and the n behind each cell

When some values are missing, the engine used in this course works out each cell from the rows where both of its variables are present, which is pairwise deletion. Each cell can then rest on a different set of rows and a different n, so cells are less comparable, and in bad cases the table as a whole can break the bound above. Report the n of each cell (the Explore widget shows it when you choose one), or use only rows complete on every variable, listwise deletion, if few are lost. Ana’s log has no missing values, so every cell of her matrix uses all 612 days.

Pooled groups and curves

A matrix pools whatever rows you give it, and a matrix of pooled data can mislead. Footfall and sales correlate at 0.89 over both cafés but at 0.98 inside Harbourside and 0.97 inside Millbrook: Millbrook has more customers with smaller tickets, so pooling the cafés blurs a nearly straight line into two. Peak barista hours and peak wait correlate at 0.53 pooled and 0.42 inside each café. The groups-and-simpsons-paradox lesson shows a pair whose sign flips. Before trusting a pooled matrix, compute it within each group too.

A matrix also holds only linear association (or monotonic association, for Spearman). Peak waits rise steeply once the load passes about seven transactions per barista hour, and rain is zero on 209 of the 306 days; a single number in a cell can’t show either shape. A near-zero cell means “no straight-line link”, not “no link”.

Building one in Excel

The Analysis ToolPak’s Correlation tool (Data > Data Analysis > Correlation) takes a block of columns with labels and returns a correlation matrix; Microsoft’s page describes each entry as CORREL’s value for that pair, and the Covariance tool beside it as doing the same with COVARIANCE.P. In use, the tool fills the lower triangle with 1s on the diagonal, writes values rather than formulas (so rerun it when the data change) and gives no p-values.

With formulas, each cell is a CORREL. With the data in B2:E31 and the variable names along the top of a grid, the cell for row variable B and column variable C is =CORREL($B$2:$B$31,C$2:C$31): the dollar signs fix the row variable’s range while the column variable’s range moves as you fill right. One more column of tests is =T.DIST.2T(ABS(r)*SQRT(n-2)/SQRT(1-r^2),n-2) with r and n replaced by their cells, the p-value of the testing-correlation lesson’s t test. Excel has no function for the adjustments below, but each is one or two columns of formulas.

Controlling the family-wise error rate: Bonferroni and Holm

The family-wise error rate is the chance of at least one false alarm among the m tests. The Bonferroni correction keeps it at or below α by testing each hypothesis at $\alpha / m$: for 28 pairs, $0.05 \div 28 = 0.00179$. Equivalently, multiply each p-value by m (capped at 1) and compare the adjusted p-value with α. It needs no assumption about how the tests depend on each other, which suits a matrix, whose tests share variables. Its cost is power: with many tests, real but moderate correlations fail the strict threshold.

Holm’s method (Holm, 1979) keeps the same guarantee and never rejects fewer hypotheses. Sort the p-values from smallest to largest, $p_{(1)} \le p_{(2)} \le \dots \le p_{(m)}$. Compare the smallest with $\alpha/m$, the next with $\alpha/(m-1)$, and so on, the k-th with $\alpha/(m-k+1)$. Reject while each p-value is below its threshold and stop at the first that isn’t: it and every larger p-value stand. The first comparison is Bonferroni’s; after that each threshold is a little less strict, because once a hypothesis is rejected there is one fewer that could be a false alarm. Holm proved that the guarantee holds whatever the dependence between the tests.

Controlling the false discovery rate: Benjamini–Hochberg

When a matrix is used to screen for leads rather than to confirm findings, guarding against even one false alarm is often too strict. Benjamini and Hochberg (1995) proposed controlling a different quantity, the false discovery rate: the expected proportion of false alarms among the hypotheses rejected (0 if none is rejected). At q = 0.05, about one in twenty of the pairs you report as discoveries may be false, on average.

The Benjamini–Hochberg procedure sorts the p-values in the same way and compares the k-th smallest with $k\,q/m$, a threshold that rises in a straight line. Find the largest k whose p-value is at or below its threshold, and reject that hypothesis and every one with a smaller p-value, even if some of those were above their own thresholds. It works from the largest p-value down, so it is called a step-up procedure; Holm’s works from the smallest up and is a step-down. Benjamini and Hochberg proved the control for independent tests. The statsmodels documentation, whose procedures this course’s engine follows, notes that most of these procedures are also robust when the tests are positively correlated. Tests in a matrix need not all be, since they share variables in both directions, so when a guarantee matters Holm’s method (or the more conservative Benjamini–Yekutieli variant, which statsmodels also offers) is the safe choice.

Which to use

• Confirming a few planned relationships. When a report will act on each finding, control the family-wise error rate with Holm’s method; it is never worse than Bonferroni.
• Screening many pairs for leads. When a matrix is a first look and every lead will be checked again, control the false discovery rate with Benjamini–Hochberg, and say that a share of the leads may be false.
• Defining the family. The adjustment depends on m, so decide before looking which tests form the family: the 28 cells of one matrix, not only the cells that looked interesting. Running a Pearson matrix, then a Spearman matrix, then one for each café and each month multiplies the tests again; an adjustment within each matrix doesn’t correct for that p-hacking.
• Large samples. With hundreds of rows almost every cell is significant with or without adjustment, so judge the size of each correlation and its interval instead.