Groups and Simpson’s paradox · B05 · Sunflower Intelligence

Before you start

You will be able to

Calculate a correlation for pooled data and within each group, and recognise Simpson’s paradox when the two disagree in sign.
Explain a reversal by a third variable whose groups differ in both x and y, using group means and a scatterplot coloured by group.
Calculate the within-group and between-group parts of a sum of products, and the pooled within-group correlation.
Distinguish an ecological correlation between group averages from a correlation between individuals, and recognise the ecological fallacy.
Decide which correlation answers a given question and write up grouped data with both the within-group and pooled results.

Review first

Covariance and Pearson’s r — Covariance as a sum of products of deviations from the means, which this lesson splits into parts.
Scatterplots — Plotting one group at a time and colouring points by group.
Outliers and influence — How a few points far from the rest can move r; here whole groups do.
Interpreting results: charts, comparisons and causes — Simpson’s paradox and confounding variables, met there with proportions.

Fictional case A fictional coffee business founded by Ana Lim. It starts as a catering and barista-training service, opens its own cafés and an online shop for beans, and plans a canned cold brew.

The idea

Fictional case

Ana wonders whether busy days are good for the average ticket (sales divided by transactions). If busier mornings mean rushed customers who buy only a coffee, she might push pastries harder on quiet days; if busy days bring groups who add food, she might do the opposite. Across all 612 café-days, footfall and the average ticket have r = −0.324: busier days go with smaller tickets.

The same data, split by café

Before acting, Ana splits the data by café. At Harbourside, footfall and the average ticket have r = 0.735; at Millbrook, r = 0.517. In both cafés, busier days go with larger tickets. Pooled, the relationship is negative. The two answers can’t both describe what happens on a busy day, and the reason is in the cafés’ averages.

Footfall and average ticket by café, March–December 2027
Café	Days	Mean footfall	Mean ticket	Ticket range	r (within café)
Harbourside	306	139.7	$5.34	$4.34–$7.20	0.735
Millbrook	306	226.3	$3.27	$2.41–$4.09	0.517
Both, pooled	612	183.0	$4.31	$2.41–$7.20	−0.324

Millbrook, in the office district, has far more people through the door, and most of them buy a coffee on the way to work. Harbourside, on the waterfront, has fewer visitors who more often sit down with food. Every one of Millbrook’s 306 days has a smaller average ticket than every one of Harbourside’s. Pool the two, and the comparison that dominates is café against café, many visitors and small tickets against few visitors and large ones, which points downwards. Inside each café, busier days go with larger tickets (the case builds this in: busy days bring more groups and more add-ons).

This is Simpson’s paradox: a relationship that holds within each of several groups disappears or reverses when the groups are combined. B01 met it with proportions; here it appears in a correlation. It is not a trick of arithmetic or an error in the data. It can arise when a third variable, here the café, differs between groups in both x and y.

Averages are not individuals

Head office sees only monthly summaries. Averaging each café’s footfall and ticket by month gives 20 points, and their correlation is −0.797, much stronger than −0.324 for the days. Inside each café the monthly averages correlate positively (0.950 at Harbourside, 0.744 at Millbrook), but the gap between the cafés dominates even more once the daily noise has been averaged away.

Each café’s ten monthly averages of footfall and average ticket. Within each café, busier months have larger tickets; across the two cafés, the busier café has the smaller tickets, so the 20 points together run downwards.

A correlation computed between group averages is an ecological correlation. It answers a question about groups (do busier café-months have smaller tickets?), and it can differ from the correlation between individuals in size and even in sign. Reading an ecological correlation as if it described each individual is the ecological fallacy.

In depth

Why combining groups can reverse a correlation

Covariance, and so r, is built from products of deviations from the overall means. When the data come in groups, each deviation splits into two parts: how far the point is from its own group’s mean, and how far its group’s mean is from the overall mean. The sum of products splits the same way, with no cross term left over:

\sum_i (x_i - \bar{x})(y_i - \bar{y}) = \underbrace{\sum_g \sum_{i \in g} (x_i - \bar{x}_g)(y_i - \bar{y}_g)}_{\text{within groups}} + \underbrace{\sum_g n_g (\bar{x}_g - \bar{x})(\bar{y}_g - \bar{y})}_{\text{between groups}}

For footfall and the average ticket, the total sum of products is −18,684.6. The within-café part is +8,845.3: inside each café, busier days go with larger tickets. The between-café part is −27,529.9: Millbrook’s mean footfall is 43.3 people above the overall mean and its mean ticket $1.04 below, Harbourside’s the reverse, and with 306 days each these two terms outweigh everything inside the cafés. The pooled correlation is negative because the between-groups part wins.

The within part, divided by the within-group sums of squares, gives the within-group correlation: $r_W = 8{,}845.3 \div \sqrt{3{,}491{,}519 \times 58.73} = 0.618$ . It is the correlation of days after each café’s own means have been subtracted, a single answer to “inside a café, do busier days have larger tickets?”. It lies between the two cafés’ own correlations, 0.517 and 0.735.

It works in both directions

Pooling can strengthen a correlation as well as reverse it. Inside Millbrook, footfall and the average ticket have r = 0.247 on weekdays and 0.128 at weekends. Together, Millbrook’s 306 days give 0.517, stronger than either. Weekdays are both busier (272 visitors on average against 110) and have slightly larger tickets ($3.35 against $3.05), so the gap between weekdays and weekends lines up with the relationship inside them and adds to it. Each weekday and weekend group also spans a narrower range of footfall, which shrinks its own r, as the previous lesson showed.

Pooling can also create a correlation where there is none inside any group, when groups differ in both variables. And splitting can go on: weekday and weekend inside each café, or month inside each of those. There is no “true” level of splitting; the right level is set by the question being asked.

Ecological correlations

Averaging within groups removes the variation inside them, so an ecological correlation usually looks stronger than the correlation of the individuals behind it: the 20 café-month averages give −0.797 against −0.324 for the 612 days. Temperature and iced share show the same effect without any reversal: 0.940 across café-days, 0.986 across café-month averages. A strong ecological correlation is accurate about groups and says much less about the individuals in them.

The classic warning is Robinson’s (1950) analysis of the 1930 US census, as reported by Subramanian and colleagues (2009). Across individuals, being born abroad and being illiterate (unable to read and write; aged 10 and over) had a small positive correlation (r = 0.12); across states, the share of residents born abroad and the share illiterate (unable to read and write; aged 10 and over) had a negative correlation (r = −0.53). States differed in many ways besides the share of residents born abroad, and inferring individual behaviour from the state figures would have got the sign wrong.

Ecological correlations are not useless. They are the right tool for questions about groups (should Lantern open its next café in an office district or on the waterfront?) and often the only data available. The error is in moving from groups to individuals without saying so.

Finding the groups

Groups are not always labelled in the data you are handed. Three places to look: categorical columns that are already there (café, region, channel, customer type, weekday or weekend); time (months, seasons, before and after a price change, which can act like groups); and the scatterplot itself, where separate clouds, or a cloud with a gap in it, often mean a grouping that nobody recorded. In Lantern’s log the café column is there, but a pooled scatterplot of footfall against ticket would show two clouds even without it.

When you find a grouping, compare three things: the group means of x and y, the correlation inside each group, and the pooled correlation. If the group means line up with the within-group pattern, pooling strengthens it; if they run against it, pooling weakens or reverses it; if they differ on only one variable, pooling dilutes it. The decomposition in the formula card turns that comparison into numbers.

Choose the grouping before you compute, from what you know about the business, not by trying splits until one gives an interesting answer. Splitting the log by café is obvious to anyone who knows Lantern has two very different cafés; splitting it by the day of the month would be a search for patterns, with the many-tests problem that module 4 deals with.

Which correlation answers the question?

A question about days, customers or transactions inside a unit (“on a busy morning at a given café, does the ticket go up?”): use the within-group correlations, each café’s own or the pooled within-group r.
A question about the units themselves (“do busier cafés have smaller tickets?”): use the group means, and say how many groups there are. With two cafés there are only two points, which can’t support a correlation at all.
A question about the whole business (“across all our café-days, what goes with what?”): the pooled correlation describes it, but report the group means and within-group correlations beside it, because the pooled r mixes two different relationships.

The groups that matter are not always recorded. A reversal needs a third variable that differs between groups; if that variable isn’t in the data, no amount of splitting can find it. When the groups are known, partial correlation (module 4) offers another way to hold a third variable fixed, and the correlation-and-causation lesson discusses confounding in general.

Formulas

Within-group and between-group sums of products

\sum_i (x_i - \bar{x})(y_i - \bar{y}) = \sum_g \sum_{i \in g} (x_i - \bar{x}_g)(y_i - \bar{y}_g) + \sum_g n_g (\bar{x}_g - \bar{x})(\bar{y}_g - \bar{y})

$x_i, y_i$: Observation i’s valuesunits of x, y
$\bar{x}, \bar{y}$: Overall means of all observationsunits of x, y
$\bar{x}_g, \bar{y}_g$: Means of group gunits of x, y
$n_g$: Number of observations in group g
$g$: The groups (cafés, stores, regions …)

Assumes. An identity: it holds for any data split into groups.

Divide each part by n − 1 to split the covariance. When the between-groups part has the opposite sign and is larger, the pooled correlation reverses.

Pooled within-group correlation

r_W = \dfrac{\sum_g \sum_{i \in g} (x_i - \bar{x}_g)(y_i - \bar{y}_g)}{\sqrt{\sum_g \sum_{i \in g} (x_i - \bar{x}_g)^2 \;\sum_g \sum_{i \in g} (y_i - \bar{y}_g)^2}}

$r_W$: Correlation within groups, with each group’s own means removed
$x_i, y_i$: Observation i’s valuesunits of x, y
$\bar{x}_g, \bar{y}_g$: Means of the group observation i belongs tounits of x, y
$g$: The groups

It is Pearson’s r of the deviations from the group means. It summarises the groups’ own correlations in one figure; still report each group’s r if they differ.

Ecological correlation

r_{\text{eco}} = r\left(\bar{x}_g,\; \bar{y}_g\right)

$r_{\text{eco}}$: Pearson’s r computed across groups
$\bar{x}_g, \bar{y}_g$: Each group’s mean of x and of yunits of x, y

Assumes. Describes groups. It need not equal, or even share a sign with, the correlation between individuals.

Usually stronger than the individual-level r, because averaging removes the variation inside groups.

Worked examples

A first example: Two bookshops, one reversal

Fictional figures

Tidewater Books has a city-centre branch and an airport branch. For six days at each, it recorded footfall and the average basket. City: footfall 310, 325, 340, 355, 370, 385; basket $12.10, $12.60, $12.40, $13.30, $13.50, $13.90. Airport: footfall 110, 125, 140, 155, 170, 185; basket $24.80, $25.90, $25.50, $26.60, $27.00, $27.60. Does a busier day go with a bigger or smaller basket?

Within each branch: city r = 0.958, airport r = 0.956. In both, busier days go with bigger baskets.
Pooled, the twelve days give r = −0.934: busier days go with much smaller baskets.
Group means: the city branch averages 347.5 visitors and a $12.97 basket; the airport 147.5 visitors and a $26.23 basket. The busier branch has the far smaller basket, and with only six days each and little spread inside each branch, that gap dominates the pooled figure.

Answer. At each branch, busier days go with bigger baskets (r ≈ 0.96). The pooled r of −0.934 compares the branches, not days.

What it means. For staffing a branch on a busy day, the within-branch result is the relevant one. The pooled result answers a different question, whether the busier branch sells smaller baskets, and with two branches that is a comparison of two numbers, not a correlation. Travellers buying gifts and city commuters buying a paperback are different customers: the branch is the third variable.

A harder example: Splitting Lantern’s sum of products

Fictional figures

Across 612 café-days, footfall and the average ticket have r = −0.324, while each café’s own r is positive. Split the sum of products into within-café and between-café parts, and calculate the pooled within-café correlation.

Means: overall footfall 183.0 and ticket $4.31; Harbourside 139.7 and $5.34; Millbrook 226.3 and $3.27.
Between cafés: $306 \times (139.7 - 183.0) \times (5.34 - 4.31) + 306 \times (226.3 - 183.0) \times (3.27 - 4.31)$ with unrounded means is −27,529.9. Each café contributes about half: one is below average on footfall and above on ticket, the other the reverse, so both products are negative.
Within cafés: the sums of products of deviations from each café’s own means are +5,240.6 at Harbourside and +3,604.8 at Millbrook, +8,845.3 in all.
Total: 8,845.3 − 27,529.9 = −18,684.6, the sum of products for all 612 days around the overall means. The between part is about three times the within part, and opposite in sign.
Within-café correlation: the within-café sums of squares are 3,491,519 for footfall and 58.73 for the ticket, so $r_W = 8{,}845.3 \div \sqrt{3{,}491{,}519 \times 58.73} = 0.618$ .

Answer. Sum of products: within cafés +8,845.3, between cafés −27,529.9, total −18,684.6. The pooled within-café correlation is 0.618.

What it means. The negative pooled r comes entirely from the difference between the cafés. Inside a café, footfall and the average ticket move together (r_W = 0.618), and that is the figure for questions about busy days. The split also shows what would make the paradox vanish: if the two cafés had the same mean ticket, the between part would be zero.

Applied: a business decision: What should Ana tell the managers?

Fictional figures

Head office’s monthly dashboard shows each café’s average footfall and ticket by month; across the 20 café-months they have r = −0.80. A consultant tells Ana: “Crowds kill your ticket. On busy days, push pastries harder.” What should Ana conclude, and what should she report?

Identify the level: the dashboard’s 20 points are café-month averages, so −0.80 is an ecological correlation, about groups of days.
Look inside the groups: within each café the monthly averages have r = 0.95 (Harbourside) and 0.74 (Millbrook), and the daily data r = 0.735 and 0.517. Busier months and busier days go with larger tickets inside each café.
Find the third variable: the café. Millbrook’s customers buy smaller tickets and come in larger numbers; the negative figure compares the cafés.
Write it down: “Inside each café, busier days go with larger average tickets (r = 0.73 at Harbourside, 0.52 at Millbrook, 2027). Across cafés, Millbrook is busier and has smaller tickets, which is why the combined monthly figures run the other way (r = −0.80).”

Answer. The consultant has committed the ecological fallacy. Inside each café, busy days go with larger tickets, not smaller ones; the −0.80 compares Millbrook with Harbourside.

What it means. There is no evidence that crowds lower the ticket on a given day, so a busy-day pastry push can’t be justified by this figure. A sensible next step is to test a pastry offer on a few days at each café and compare tickets, which the correlation-and-causation lesson discusses. And dashboards that mix cafés should show each café separately, or at least label which comparison a figure makes.

Explore

Common mistakes

“The correlation across all our data is the real one; splitting is cherry-picking.”
Pooled data mix the relationship inside groups with the difference between them. When groups differ in both variables, the pooled r can reverse the relationship inside every group. Report both and say which answers the question; splitting by a variable chosen before looking at the results is good practice, not cherry-picking.
“Busier cafés have smaller tickets, so busy days lower the ticket.”
That moves from groups to individuals: the ecological fallacy. Across the cafés the busier one has smaller tickets; inside each café, busier days have larger ones. Answer questions about days with day-level data inside each café.
“With 20 monthly points, r = −0.80 is strong evidence.”
An ecological correlation is usually stronger than the individual-level one because averaging removes the variation inside groups, and here it is driven by two cafés. Twenty averages from two cafés are closer to two points than to twenty independent observations.
“Simpson’s paradox only happens with percentages.”
It happens with correlations too, as Lantern’s cafés show: footfall and the average ticket rise together inside each café and run the other way when the cafés are pooled. When a third variable differs between groups in both x and y, pooling can reverse, create or inflate a correlation.
“If I split by enough variables, I will find the true correlation.”
Every split answers a different question, and splitting by many variables in search of a pattern is a multiple-testing problem (module 4). Choose the grouping from the question and from what is known about the business, before computing.

Guided practice

Work through these with hints when you need them. Every hint and every worked solution is there to be used.

Loading three questions…

On your own

No hints this time. Answer, check, and read the feedback; the solution appears once you have answered or if you ask for it. Your first answers count toward mastery.

Loading five questions…

In practice

Fictional case

Ana’s note to her managers: “Inside each café, busier days go with larger average tickets: r = 0.73 at Harbourside and 0.52 at Millbrook across 2027. Millbrook is busier and has smaller tickets, so the two cafés together show the opposite (r = −0.32). Please don’t read the combined monthly dashboard as a statement about busy days; from next month it shows each café on its own line.”

Her note shows the habits that keep grouped data honest. She leads with the correlation that answers the managers’ question, which is about days inside their own café. She gives the pooled figure too, so nobody who sees it elsewhere thinks it was hidden, and she names the third variable that explains the difference. And she fixes the dashboard so that the misleading comparison doesn’t keep arriving every month.

The same structure turns up across business analytics. Store-level data pool stores of different sizes and locations; customer data pool segments that differ in income and habit; data across years pool periods with different prices. A chain whose larger stores have lower margins may find that within every store, busier weeks have higher margins; a lender whose regions with more applications approve fewer may find the opposite inside each region. In each case the first step is the same: plot by group, compute within-group and pooled correlations, and compare the group means.

When summaries are all you have, say so. A correlation between regional averages, monthly figures or store totals is a statement about regions, months or stores. It can guide decisions about those units, such as where to open, what to stock, which months to staff, but it should not be repeated as a fact about individual customers or days without data at that level.

Knowledge check

Six questions, one attempt each, no hints. Score 80% or more to complete the lesson; you can take it again at any time.

Loading six questions…

Summary

Pooled data can show the opposite of every group: footfall and average ticket have r = −0.324 across both cafés but 0.735 at Harbourside and 0.517 at Millbrook. This is Simpson’s paradox.
It happens when a third variable (the café) differs between groups in both x and y: Millbrook has more visitors (226 against 140) and smaller tickets ($3.27 against $5.34).
The sum of products splits exactly into within-group and between-group parts (+8,845.3 and −27,529.9 here); the pooled r is negative because the between part wins. The pooled within-group r is 0.618.
Pooling can also create or strengthen a correlation: Millbrook’s weekdays (0.247) and weekends (0.128) combine to 0.517.
A correlation between group averages is ecological (−0.797 across 20 café-months). It describes groups, is usually stronger than the individual-level r, and can differ in sign; reading it as a fact about individuals is the ecological fallacy.
Choose the correlation that matches the question, plot by group before pooling, and report within-group and pooled results together with the group means.

Key formulas

Within-group and between-group sums of products
Pooled within-group correlation
Ecological correlation

Terms introduced

Within-group correlation: The correlation between two variables measured inside groups, after each group’s own means have been subtracted, so that differences between the groups play no part.
Ecological correlation: A correlation computed between group averages or rates, such as cafés’ monthly means or regions’ percentages, rather than between individual observations.
Ecological fallacy: The error of assuming that a characteristic of a group or area, such as a neighbourhood’s geodemographic type or average income, holds for each individual in it.
Pooled data: Observations from several groups combined into one dataset and analysed together, ignoring which group each came from.

Sources

Malinas, G. & Bigelow, J. (2004). “Simpson’s Paradox”. Stanford Encyclopedia of Philosophy (Spring 2006 Edition), ed. E. N. Zalta. plato.stanford.edu/archives/spr2006/entries/paradox-simpson
Subramanian, S. V., Jones, K., Kaddour, A. & Krieger, N. (2009). “Revisiting Robinson: the perils of individualistic and ecologic fallacy”. International Journal of Epidemiology, 38(2), 342–360. pmc.ncbi.nlm.nih.gov/articles/PMC2663721
Holmes, A., Illowsky, B. & Dean, S. (2023). Introductory Business Statistics 2e, section 13.1, “The Correlation Coefficient r”. OpenStax. openstax.org/books/introductory-business-statistics-2e/pages…

Before you start

You will be able to

Review first

The idea

The same data, split by café

Averages are not individuals

In depth

Why combining groups can reverse a correlation

It works in both directions

Ecological correlations

Finding the groups

Which correlation answers the question?

Formulas

Within-group and between-group sums of products

Pooled within-group correlation

Ecological correlation

Worked examples

Explore

Common mistakes

“The correlation across all our data is the real one; splitting is cherry-picking.”

“Busier cafés have smaller tickets, so busy days lower the ticket.”

“With 20 monthly points, r = −0.80 is strong evidence.”

“Simpson’s paradox only happens with percentages.”

“If I split by enough variables, I will find the true correlation.”

Guided practice

On your own

In practice

Knowledge check

Summary

Key formulas

Terms introduced

Sources