Partial correlation · B05 · Sunflower Intelligence

Before you start

You will be able to

Explain what it means to hold a third variable fixed, and compare days that are alike on it to see the idea at work.
Calculate a partial correlation by correlating residuals and with the first-order formula, in Excel and by hand, and test it with n − 2 − k degrees of freedom.
Interpret a partial correlation that differs in size or sign from the simple correlation, without claiming more than it shows.
Distinguish confounders, mediators and colliders, and explain why controlling for a mediator or a collider misleads.
Identify the limits of partial correlation: straight-line control only, measurement error in the control, and variables left out.

Review first

Correlation matrices — The matrix in which peak barista hours and peak waits correlate at +0.53, the puzzle this lesson resolves.
Testing a correlation — The t test of a correlation, which a partial correlation reuses with fewer degrees of freedom.
Curves, thresholds and restricted ranges — The steep rise in waits once the load passes about seven transactions per barista hour.
Descriptive statistics and regression in Excel — SLOPE, INTERCEPT and the least-squares line, used here only to take out a third variable’s straight-line part.

Fictional case A fictional coffee business founded by Ana Lim. It starts as a catering and barista-training service, opens its own cafés and an online shop for beans, and plans a canned cold brew.

The idea

Fictional case

Ana’s correlation matrix shows peak barista hours and the peak wait correlating at +0.53 over both cafés’ 612 days. Read at face value, days with more baristas at the 7–11 a.m. peak had longer waits, and a manager under pressure to cut costs might conclude that extra baristas get in each other’s way. Ana suspects something else: both numbers rise on busy days. More customers mean longer queues, and her managers roster more baristas on days they expect to be busy.

Comparing like with like

The way to check is to compare days that were equally busy. If the extra baristas are only a sign of a busy day, then among days with about the same number of transactions the link should disappear; if baristas shorten queues, it should turn negative. The table splits the 612 days by transactions and finds the correlation of peak barista hours with the peak wait inside each band.

Peak barista hours and peak wait within bands of daily transactions (both cafés, 612 days)
Transactions	Days	Mean peak wait (min)	r (barista hours, wait)
0–99	371	1.99	−0.13
100–149	126	3.25	−0.58
150–199	105	5.54	−0.59
200–299	10	9.04	−0.44

Inside every band the correlation is negative: −0.58 and −0.59 on days with 100 to 199 transactions, where queues build, and only −0.13 on quiet days, when the wait is about two minutes whatever the roster. Among equally busy days, more barista hours go with shorter waits. The positive correlation over all days came from mixing quiet days, with few baristas and short waits, with busy days, with more baristas and longer waits. This is the same mechanism as the groups-and-simpsons-paradox lesson, with a continuous variable instead of two cafés.

Bands are a rough tool. They throw away information (days with 101 and 149 transactions are treated alike), the bands can be few days each, and with two or three variables to hold fixed the bands multiply fast. Partial correlation does the same job with all the data at once.

Holding transactions fixed with residuals

Draw the least-squares line through the scatterplot of peak barista hours against transactions. Each day’s residual is its barista hours minus the line’s value at its number of transactions: positive on days rostered more heavily than usual for that much trade, negative on days rostered lightly. Do the same for the peak wait: a positive residual means a longer wait than usual for that many transactions. Both residuals have had transactions’ straight-line part taken out, so their correlation is the link between barista hours and waits with transactions held fixed. That correlation is the partial correlation.

For the 612 days it is $r_{SW \cdot T} = -0.725$ , where S is peak barista hours, W the peak wait and T transactions, and the dot is read “holding fixed”. The sign has flipped: on days with more barista hours than usual for their trade, waits were shorter than usual for their trade. The Explore widget plots the two sets of residuals; switch it to the raw values to see the +0.53 cloud again.

The first-order formula

With one variable held fixed, the partial correlation can be worked out from the three simple correlations, so a correlation matrix is enough:

r_{SW\cdot T} = \dfrac{r_{SW} - r_{ST}\,r_{WT}}{\sqrt{(1 - r_{ST}^2)(1 - r_{WT}^2)}} = \dfrac{0.535 - 0.926 \times 0.767}{\sqrt{(1 - 0.926^2)(1 - 0.767^2)}} = \dfrac{-0.175}{0.242} \approx -0.725

The numerator subtracts from $r_{SW}$ the part that both variables’ links with transactions would produce on their own, $r_{ST} \times r_{WT} = 0.710$ . The simple correlation, 0.535, is less than that, so once transactions are accounted for the remaining link is negative. The denominator rescales the result to the −1 to 1 range, using what is left of each variable after transactions are taken out. It is called first-order because it holds one variable fixed; a simple correlation is zero-order.

The case was built this way. The generator lets the peak wait depend on transactions per barista hour, so more baristas shorten the wait at a given level of trade, and lets the managers roster for the trade they expect. The partial correlation recovers the planted direction from the log; the simple correlation points the wrong way. With real data no one tells you what was planted, which is why the correlation-and-causation lesson asks what more evidence a causal claim needs.

In depth

Definition and notation

The partial correlation of x and y holding z fixed, $r_{xy\cdot z}$ , is the correlation between the residuals of x after a least-squares fit on z and the residuals of y after a least-squares fit on z. With several variables held fixed, $z_1, \dots, z_k$ , fit each of x and y on all of them at once (multiple regression, B06) and correlate the residuals: $r_{xy\cdot z_1 \dots z_k}$ is of order k. The first-order formula gives the same number as the residual method when one variable is held fixed. Like any correlation it lies between −1 and 1.

Ana can hold the café fixed as well as transactions, as a 0/1 variable (1 for Millbrook). The second-order partial correlation is −0.719, almost unchanged from −0.725, so the difference between the cafés was not what made the simple correlation positive. Inside each café separately, with transactions held fixed, it is −0.54 at Harbourside and −0.83 at Millbrook: the direction agrees, the strength differs.

Testing a partial correlation

To test $H_0: \rho_{xy\cdot z} = 0$ , use the testing-correlation lesson’s t statistic with k fewer degrees of freedom, one for each variable held fixed: $t = r\sqrt{(n - 2 - k) / (1 - r^2)}$ on $n - 2 - k$ df (Penn State’s STAT 505 notes, section 6.3, give the same rule). For the 612 days, $r = -0.7246$ and $t = -0.7246\sqrt{609 / (1 - 0.7246^2)} \approx -25.9$ on 609 df, and the two-sided p-value is below 0.001. In Excel, =T.DIST.2T(ABS(t),n-2-k). The interval and power ideas of the testing-correlation lesson carry over the same way.

Partial correlation in Excel

There is no PARTIAL function. Two routes give the same answer for one control. The formula route: three CORREL cells and the first-order formula, =(B1-B2*B3)/SQRT((1-B2^2)*(1-B3^2)) with r_xy, r_xz and r_yz in B1:B3. The residual route: a column of residuals for each variable, =C2-(INTERCEPT(C$2:C$32,$D$2:$D$32)+SLOPE(C$2:C$32,$D$2:$D$32)*$D2) with the control in column D, filled down and across, then CORREL of the two residual columns. The residual route shows each day’s residual, so you can plot them and spot the days that drive the result, and it extends to several controls with LINEST.

What a partial correlation can’t do

It removes only the straight-line part of the control. Waits rise along a curve as trade grows (a threshold effect), and the bands show the link between baristas and waits is weak on quiet days and strong on busy ones. The single partial correlation, −0.72, averages over that. Partial correlation on ranks (−0.47 here) or a comparison within bands can show what the straight line misses.
It can’t hold fixed what wasn’t measured. Anything that affects waits apart from the number of transactions, and also goes with the roster, is still in the comparison. A large catering order that ties up a barista at the peak, or an espresso machine that breaks down, lengthens waits without adding transactions; if managers tend to roster extra help on those mornings, holding transactions fixed doesn’t remove it.
A control measured with error controls only partly. If the door counter misses some customers, holding footfall fixed leaves some of the busy-day effect in, a problem called residual confounding.
It is still a correlation. A partial correlation says how two variables go together at given values of a third; whether one causes the other needs the evidence of the next lesson.

Which variables to hold fixed

The arithmetic will hold anything fixed. Whether it should depends on how the variables are connected, which the data alone can’t tell you; it comes from knowing how the business works and the order in which things happen. Three cases matter.

Three ways a third variable can be connected: a confounder and a mediator on the first row, a collider below. Arrows show what affects what. Hold a confounder fixed; don’t hold a mediator fixed if you want the whole link; never hold a collider fixed.

A confounding variable affects both variables, as a busy day drives both the roster and the queue. Holding it fixed removes the link it creates, which is what Ana did with transactions.

A mediator lies on the path from one variable to the other: x affects m, and m affects y. Holding a mediator fixed removes the part of the link that runs through it, so the partial correlation measures only what is left, not the whole effect. At Harbourside warm days bring more people to the waterfront, and more people buy more; the case’s generator lets the temperature reach sales only through footfall. Leaving out the five festival days, whose crowds have nothing to do with the weather, the high temperature and sales correlate at 0.49 over the other 301 days; holding footfall fixed gives −0.02. The 0.49 answers “do warm days go with higher sales?”, and the answer is yes. The −0.02 answers a narrower question, “at a given footfall, do warm days sell more?”, and the answer is no, which is what a mediator predicts: once footfall is fixed, the route is closed. (With the festival days in, the partial correlation is −0.19, because sales rise along a curve in footfall and the festivals sit at its far end; a straight-line control can’t remove that, which is the first limit listed above.) Within the case the peak load is the starkest example: it is transactions per barista hour, the route through which baristas shorten waits, and holding it fixed leaves barista hours and waits correlating at +0.07, as if baristas didn’t matter.

A collider is affected by both variables. Holding a collider fixed creates a link between its causes even if they are unrelated, because among days with the same value of the collider, a high value of one cause must be offset by the other (Cole and colleagues, 2010, give clear examples). At Harbourside the high temperature and rain are almost unrelated (r = 0.05), but warm days raise footfall and rain lowers it. Holding footfall fixed gives a partial correlation of 0.17 (p = 0.004): among days with the same footfall, the warmer ones were more often wet, because a warm, dry day would have drawn more people. The link is created by the analysis, not by the weather.

Formulas

First-order partial correlation

r_{xy\cdot z} = \dfrac{r_{xy} - r_{xz}\,r_{yz}}{\sqrt{(1 - r_{xz}^2)(1 - r_{yz}^2)}}

$r_{xy\cdot z}$: Correlation of x and y holding z fixed
$r_{xy}$: Simple correlation of x and y
$r_{xz}, r_{yz}$: Correlations of x and of y with the control z

Assumes. All three correlations from the same rows; z’s links with x and y are roughly straight lines.

Undefined if z correlates perfectly with x or y. Gives the same value as correlating residuals.

Partial correlation as a correlation of residuals

r_{xy\cdot z} = \operatorname{corr}(e_x, e_y), \quad e_x = x - (a_x + b_x z), \quad e_y = y - (a_y + b_y z)

$r_{xy\cdot z}$: Partial correlation of x and y holding z fixed
$e_x, e_y$: Residuals of x and of y after a least-squares line on zunits of x, of y
$a_x, b_x$: Intercept and slope of the line of x on z (INTERCEPT, SLOPE)units of x; of x per unit of z
$a_y, b_y$: Intercept and slope of the line of y on zunits of y; of y per unit of z
$x, y, z$: The two variables of interest and the controltheir units

With several controls, fit x and y on all of them (LINEST or multiple regression, B06) and correlate the residuals.

Test of a partial correlation

t = r_{xy\cdot z}\sqrt{\dfrac{n - 2 - k}{1 - r_{xy\cdot z}^2}}, \qquad \text{df} = n - 2 - k

$r_{xy\cdot z}$: The partial correlation
$n$: Number of rows complete on every variable
$k$: Number of variables held fixed
$t$: Test statistic for H₀: the partial correlation is 0
$\text{df}$: Degrees of freedom

Assumes. As for the test of a simple correlation, applied to the residuals.

Excel: T.DIST.2T(ABS(t), n − 2 − k).

Worked examples

A first example: Cabin crew and complaints

Fictional figures

Swiftwing Air (fictional) looked at 50 flights. The number of cabin crew and the number of passenger complaints correlate at 0.40, which worries the operations director. Crew correlate with the number of passengers at 0.85 (bigger flights carry more crew) and complaints with passengers at 0.60. What is the correlation of crew and complaints holding passengers fixed, and is it significant at 5%?

Name the variables: x = crew, y = complaints, z = passengers; $r_{xy} = 0.40$ , $r_{xz} = 0.85$ , $r_{yz} = 0.60$ .
Numerator: $r_{xy} - r_{xz} r_{yz} = 0.40 - 0.85 \times 0.60 = 0.40 - 0.51 = -0.11$ .
Denominator: $\sqrt{(1 - 0.85^2)(1 - 0.60^2)} = \sqrt{0.2775 \times 0.64} = 0.4214$ , so $r_{xy\cdot z} = -0.11 \div 0.4214 \approx -0.261$ .
Test with one variable held fixed: df = 50 − 2 − 1 = 47, $t = -0.261\sqrt{47 \div (1 - 0.261^2)} \approx -1.85$ , and =T.DIST.2T(1.85,47) ≈ 0.070.

Answer. r(crew, complaints · passengers) ≈ −0.26, t ≈ −1.85 on 47 df, p ≈ 0.070: not significant at 5%.

What it means. The worrying +0.40 comes from flight size: big flights have more crew and more complaints. On flights of the same size there is a weak tendency for more crew to go with fewer complaints, but 50 flights are too few to be sure of it. The director should stop worrying about crew causing complaints and, if she wants to know whether extra crew reduce them, look at more flights or test it directly.

A harder example: Baristas and waits, three ways

Fictional figures

Use Ana’s full log to find the partial correlation of peak barista hours (S) and the peak wait (W) holding transactions (T) fixed, by the formula and by residuals; test it; then check whether the café, or the curve in waits, changes the conclusion.

Simple correlations from the matrix: $r_{SW} = 0.535$ , $r_{ST} = 0.926$ , $r_{WT} = 0.767$ . Formula: $(0.535 - 0.710) \div \sqrt{0.142 \times 0.412} = -0.175 \div 0.242 \approx -0.725$ .
Residuals: fitting barista hours and waits each on transactions and correlating the 612 pairs of residuals gives −0.725 as well, the same value, as it must with one control.
Test: k = 1, so df = 609 and $t \approx -25.9$ ; the two-sided p-value is below 0.001.
Café: holding the café fixed as well (a second-order partial) gives −0.719; within Harbourside alone, −0.54, and within Millbrook alone, −0.83. The sign is the same everywhere.
The curve: within bands of transactions the correlation is −0.13 on quiet days (under 100 transactions) and −0.58 to −0.59 on days with 100 to 199. On ranks the partial correlation is −0.47. The straight-line partial correlation summarises a link that is weak when the cafés are quiet and strong when they are busy.

Answer. r(S, W · T) = −0.725 (t ≈ −25.9, 609 df, p below 0.001), robust to holding the café fixed and negative in both cafés and in every band of trade, but much stronger on busy days than quiet ones.

What it means. At a given level of trade, days rostered with more barista hours had shorter peak waits, most clearly on busy days. That agrees with what the case’s generator planted (waits depend on transactions per barista hour). It is the opposite of what the simple +0.53 suggested, and it is the number Ana should bring to a roster discussion, with the caveat that the log is observational.

Applied: a business decision: Which partial correlation answers the roster question?

Fictional figures

Ana’s operations manager has run three partial correlations of peak barista hours and the peak wait over the 612 days and asks which one should guide the roster: holding transactions fixed (−0.72), holding the peak load fixed (+0.07), or holding nothing fixed (+0.53).

Write down how the variables are connected. Expected trade drives the roster and actual trade drives the queue, so transactions are a confounder of barista hours and waits. The peak load is transactions × 0.55 ÷ barista hours: it is computed from the roster itself, and it is the route by which extra baristas shorten the queue, so it is a mediator.
Holding nothing fixed (+0.53) mixes busy and quiet days and answers “were waits longer on heavily rostered days?”, which is no help for a decision.
Holding the load fixed (+0.07, p = 0.10) removes the very route through which baristas work. It answers “at the same load per barista, do more baristas go with shorter waits?”, and the answer is no, because at the same load, more baristas means more customers.
Holding transactions fixed (−0.72) compares equally busy days and keeps the baristas’ route open. It answers the roster question: on a day with a given amount of trade, do more barista hours go with shorter waits?

Answer. Use the partial correlation holding transactions fixed, −0.72. Holding nothing fixed confounds the link with busy days, and holding the load fixed controls for a mediator and hides the link.

What it means. The choice of control is a judgement about how the café works, made before looking at the numbers. Ana’s note to the operations manager should say which variable was held fixed and why, and that a planned trial (the next lesson) would show more convincingly how much each extra barista hour shortens the queue.

Explore

Common mistakes

“The partial correlation is the real correlation; the simple one was wrong.”
Both are correct answers to different questions. The simple correlation describes how barista hours and waits went together across all days; the partial correlation describes how they went together at a given level of trade. Say which question you are answering.
“I held everything fixed to be safe.”
Holding a mediator fixed removes the effect you want to measure (holding the peak load fixed makes baristas look useless); holding a collider fixed creates a link that doesn’t exist (temperature and rain, once footfall is fixed). Choose each control from what you know about the process.
“Holding transactions fixed means the busy-day effect is gone completely.”
A partial correlation removes only the straight-line part of the control’s link, and only for the control as measured. The baristas–wait link is weak on quiet days and strong on busy ones, and anything unmeasured that drives both is still there. Look at bands or ranks, and list what you couldn’t hold fixed.
“I used n − 2 degrees of freedom for the partial correlation.”
Each variable held fixed uses up one more degree of freedom: test with n − 2 − k. With 612 rows it hardly matters; with 30 rows and three controls it does.
“A negative partial correlation proves that baristas shorten waits.”
It shows that, at a given level of trade, the two went together that way. An unmeasured confounder could produce the same pattern without any effect of the extra hours: if the manager put her most experienced baristas on the mornings she rostered extra hours, their speed, not the hours, could shorten the waits. Causal claims need the evidence of the next lesson.

Guided practice

Work through these with hints when you need them. Every hint and every worked solution is there to be used.

Loading four questions…

On your own

No hints this time. Answer, check, and read the feedback; the solution appears once you have answered or if you ask for it. Your first answers count toward mastery.

Loading six questions…

In practice

Partial correlation is how analysts ask whether a link survives once an obvious common driver is accounted for. A retailer finds that stores with more staff have more shrinkage, and holds store size fixed. A hospital finds that wards with more nurses have more incidents, and holds the number and severity of patients fixed. A bank finds that customers who use its mobile app more often hold larger balances, and holds their income fixed. In each case the simple correlation mixes the link of interest with the effect of size, busyness or means, and the partial correlation compares like with like.

Fictional case

Ana’s note to her operations manager reads: “Over the 612 café-days, more peak barista hours went with longer waits (r = +0.53), because both rise on busy days. Holding the day’s transactions fixed, the link reverses: on days with the same trade, more barista hours went with shorter waits (partial r = −0.72, p below 0.001), in both cafés, and most clearly on days with more than 100 transactions. I did not hold the load per barista fixed, because it is the route by which baristas would shorten queues. This is observational, so before changing the roster I would like to trial it.”

Her note does what a good partial-correlation report does. It gives the simple and the partial correlation side by side, so the reader sees what holding transactions fixed changed. It names the control and explains why it is a confounder, and names the variable deliberately not held fixed and why. It shows where the link is strong and where it is weak rather than relying on one number. And it stops short of a causal claim, which the last lesson of the module takes up.

In regression (B06) the same idea appears as a coefficient: the slope on barista hours in a model that also contains transactions measures the change in waits per extra barista hour at a given level of trade. Partial correlation gives the strength of that link on the −1 to 1 scale; regression gives its size in minutes per hour. Both depend on the same choice of controls.

Knowledge check

Six questions, one attempt each, no hints. Score 80% or more to complete the lesson; you can take it again at any time.

Loading six questions…

Summary

A partial correlation measures the link between two variables holding one or more others fixed. With one control it is r_xy·z = (r_xy − r_xz r_yz) ÷ √((1 − r_xz²)(1 − r_yz²)).
It equals the correlation of the residuals of x and y after each is fitted on the control by least squares; comparing like with like in bands shows the same idea roughly.
Peak barista hours and waits correlate at +0.53 because both rise on busy days; holding transactions fixed the partial correlation is −0.72, the sign the case planted.
Test a partial correlation with t = r√((n − 2 − k) ÷ (1 − r²)) on n − 2 − k degrees of freedom, k being the number of controls.
Hold confounders fixed. Don’t hold a mediator fixed when you want the whole link (holding the load fixed hides the baristas’ effect), and never hold a collider fixed (it creates a link between temperature and rain).
A partial correlation removes only the straight-line part of the measured controls: curves, measurement error and unmeasured drivers remain. It is still a correlation, not proof of cause.

Key formulas

First-order partial correlation
Partial correlation as a correlation of residuals
Test of a partial correlation

Terms introduced

Partial correlation: The correlation between two variables after the straight-line part of their links with one or more other variables has been removed: the correlation of their residuals. Written r_xy·z for one control z.
Mediator: A variable on the path between two others, through which one affects the other (x affects m, and m affects y). Holding it fixed removes the part of the link that runs through it.
Collider: A variable affected by two others. Holding it fixed, or selecting on it, creates an association between its causes even when they are unrelated.
Residual confounding: The confounding that remains after adjusting for a confounder that is measured imperfectly or held fixed only roughly, such as by a straight line or in broad bands.

Sources

Penn State Eberly College of Science. STAT 505: Applied Multivariate Statistical Analysis, Lesson 6, “Multivariate Conditional Distribution and Partial Correlation” (sections 6.1–6.3). online.stat.psu.edu/stat505/book/export/html/638
Cole, S. R., Platt, R. W., Schisterman, E. F., Chu, H., Westreich, D., Richardson, D. & Poole, C. (2010). “Illustrating bias due to conditioning on a collider”. International Journal of Epidemiology, 39(2), 417–420. pmc.ncbi.nlm.nih.gov/articles/PMC2846442
Illowsky, B. & Dean, S. Introductory Statistics 2e, section 12.4, “Testing the Significance of the Correlation Coefficient”. OpenStax. openstax.org/books/introductory-statistics/pages/12-4-testin…
Microsoft Support. “CORREL function”. support.microsoft.com/en-us/excel/correl-function

Before you start

You will be able to

Review first

The idea

Comparing like with like

Holding transactions fixed with residuals

The first-order formula

In depth

Definition and notation

Testing a partial correlation

Partial correlation in Excel

What a partial correlation can’t do

Which variables to hold fixed

Formulas

First-order partial correlation

Partial correlation as a correlation of residuals

Test of a partial correlation

Worked examples

Explore

Common mistakes

“The partial correlation is the real correlation; the simple one was wrong.”

“I held everything fixed to be safe.”

“Holding transactions fixed means the busy-day effect is gone completely.”

“I used n − 2 degrees of freedom for the partial correlation.”

“A negative partial correlation proves that baristas shorten waits.”

Guided practice

On your own

In practice

Knowledge check

Summary

Key formulas

Terms introduced

Sources