Sunflower Intelligence
Business Analytics & Data Science

B05 · Intermediate

Correlation Analysis

10 lessons

B05 · Intermediate · 10 lessons in four modules · about 13 hours

Correlation Analysis

Measure how variables move together, test it, and avoid reading causation into it.

What you will learn

  • Read a scatterplot for direction, form, strength and unusual points, and choose a plot that shows what a single number hides.
  • Calculate and interpret covariance, Pearson’s r, r², Spearman’s ρ and Kendall’s τ, and choose between them for the data and the question.
  • Test a correlation, give a confidence interval for it, judge the sample size needed to detect one, and control for testing many pairs at once.
  • Find what outliers, skew, curved relationships, groups and restricted ranges do to a correlation, and handle each honestly.
  • Build and read a correlation matrix and use partial correlation to ask what remains when a third variable is held fixed.
  • Explain why correlation does not show causation (confounding, reverse causation, trends, selection, chance), and say what evidence would.

Prerequisites

How it works

Each lesson teaches the idea from first principles, works examples step by step, lets you explore it in an interactive, then has you practise with feedback. A lesson is complete when you have worked its guided problems and scored 80% on its knowledge check. Mastery is tracked separately and comes back for review before you are likely to forget. Each module ends with a test; the course ends with a project and a final assessment.

Fictional case

Lantern Coffee Co. A fictional coffee business founded by Ana Lim. It starts as a catering and barista-training service, opens its own cafés and an online shop for beans, and plans a canned cold brew. Every name and figure in it is invented for teaching.

Not sure where to start?

The diagnostic has 12 questions across the course. It doesn’t count toward anything; it suggests the module to start from. Take the diagnostic.

Modules and lessons

Module 1 Seeing relationships

  1. ScatterplotsPlot before you calculate. Read a scatterplot for direction, form, strength and unusual points, choose which variable goes on which axis, and make 612 café-days readable when points pile on top of one another, hide their time order, or tell four different stories behind one correlation.80 min · not started
  2. Covariance and Pearson’s rTurn a scatterplot into a number. Build covariance from the products of deviations, see why its size depends on the units, divide by the two standard deviations to get Pearson’s r, read r as the average product of z-scores, and interpret r² with care. Then follow footfall to transactions to sales through the cafés’ log.75 min · not started
  3. Module 1 test11 questions on this module. 70% passes.not taken

Module 2 Measuring and testing

  1. Rank correlationReplace values by their ranks and measure how consistently one variable rises (or falls) with another, whether or not the path is straight. Calculate Spearman’s ρ and Kendall’s τ-b by hand and in Excel, handle ties, and choose between r, ρ and τ for curves, data-entry slips, ratings and a rainfall column that is zero on most days.80 min · not started
  2. Testing a correlationDecide whether a sample correlation is evidence of a real association, put a confidence interval round it with Fisher’s z, and plan how many pairs a study needs. Test rain against footfall at both cafés, see why r = 0.14 is significant with 612 pairs while r = 0.64 is not with 8, and compare the two cafés’ correlations without confusing significance with size.85 min · not started
  3. Module 2 test11 questions on this module. 70% passes.not taken

Module 3 When correlations mislead

  1. Outliers and influenceOne wrong digit among 612 days can turn a near-perfect correlation into a middling one. Find the points that move a correlation, measure how much each one matters, and decide what to do with them without deleting what you dislike.75 min · not started
  2. Curves, thresholds and restricted rangesPearson’s r measures how close points come to a straight line. Learn what it says, and fails to say, about curves, thresholds and U-shapes, why looking at only part of the range shrinks it, and why noisy measurements do too.85 min · not started
  3. Groups and Simpson’s paradoxBusier days go with bigger tickets in each of Lantern’s cafés, yet the two cafés pooled say the opposite. Learn why combining groups can reverse, create or inflate a correlation, why correlations of averages are not correlations of individuals, and what to report.70 min · not started
  4. Module 3 test10 questions on this module. 70% passes.not taken

Module 4 Many variables and causation

  1. Correlation matricesPut every pair of measures in one table, read it as a map of what goes with what, spot measures that say the same thing, and test 28 correlations at once without being fooled by the false alarms that many tests produce: Bonferroni, Holm and the Benjamini–Hochberg false discovery rate.80 min · not started
  2. Partial correlationAsk what remains of a correlation when a third variable is held fixed: by comparing like with like, by correlating residuals, or with the first-order formula. Then learn which variables to hold fixed (confounders) and which not to (mediators and colliders).70 min · not started
  3. Correlation and causationLearn the six ways two variables can be correlated without one causing the other, what kinds of evidence do support a causal claim, and how to write a finding that says exactly as much as the data allow.75 min · not started
  4. Module 4 test11 questions on this module. 70% passes.not taken

Project and final assessment

  1. Project: Does paid social sell beans?Ana wants to double her paid-social budget because spend and sales are correlated. Analyse the online shop’s 52 weeks of 2027: plots, the right correlations, an outage week, ratings and delivery, trends, and what a correlation can and can’t say about her budget. Then recommend how she should find out.not taken
  2. Final assessment24 questions across the whole course. 70% passes.not taken

The course is complete when every lesson is complete, every module test is passed, the project is finished and the final assessment is passed.

Resources

  • Glossary: every term this course defines.
  • Formulas: every formula with its symbols and units.
  • Review: flashcards and the questions you missed, when they are due.

Elsewhere on this site

Sources used in this course

  1. Illowsky, B. & Dean, S. (2023). Introductory Statistics 2e, sections 12.2, “Scatter Plots”, and 12.3, “The Regression Equation”. OpenStax. openstax.org/books/introductory-statistics-2e/pages/12-2-sca…
  2. NIST/SEMATECH. e-Handbook of Statistical Methods, section 1.3.3.26, “Scatter Plot”. itl.nist.gov/div898/handbook/eda/section3/scatterp.htm
  3. Anscombe, F. J. (1973). “Graphs in statistical analysis”. The American Statistician, 27(1), 17–21. doi.org/10.1080/00031305.1973.10478966 (not yet verified)
  4. Wikipedia. “Anscombe’s quartet” (prints the four datasets and their summary statistics; the same data ship with R as datasets::anscombe). en.wikipedia.org/wiki/Anscombe%27s_quartet
  5. Microsoft Support. “CORREL function”. support.microsoft.com/en-US/excel/functions/correl-function
  6. Holmes, A., Illowsky, B. & Dean, S. (2023). Introductory Business Statistics 2e, section 13.1, “The Correlation Coefficient r”. OpenStax. openstax.org/books/introductory-business-statistics-2e/pages…
  7. Illowsky, B. & Dean, S. (2023). Introductory Statistics 2e, section 12.3, “The Regression Equation” (the correlation coefficient and the coefficient of determination). OpenStax. openstax.org/books/introductory-statistics-2e/pages/12-3-the…
  8. Microsoft Support. “COVARIANCE.S function”. support.microsoft.com/en-US/excel/functions/covariance-s-fun…
  9. NIST/SEMATECH. Dataplot reference manual, “Rank correlation independence test” (Spearman’s ρ as the correlation of ranks, with average ranks for ties). itl.nist.gov/div898/software/dataplot/refman1/auxillar/rankc…
  10. NIST/SEMATECH. Dataplot reference manual, “Kendall’s tau” (concordant and discordant pairs; τ-a, τ-b and the treatment of ties). itl.nist.gov/div898/software/dataplot/refman2/auxillar/kende…
  11. SciPy documentation. “scipy.stats.kendalltau” (τ-b as the default; ties in x, in y and in both). docs.scipy.org/doc/scipy-1.12.0/reference/generated/scipy.st…
  12. Microsoft Support. “RANK.AVG function”. support.microsoft.com/en-US/excel/functions/rank-avg-functio…
  13. Illowsky, B. & Dean, S. (2023). Introductory Statistics 2e, section 12.4, “Testing the Significance of the Correlation Coefficient”. OpenStax. openstax.org/books/introductory-statistics-2e/pages/12-4-tes…
  14. Microsoft Support. “FISHER function”. support.microsoft.com/en-US/excel/functions/fisher-function
  15. Microsoft Support. “FISHERINV function”. support.microsoft.com/en-US/excel/functions/fisherinv-functi…
  16. Illowsky, B. & Dean, S. Introductory Statistics 2e, section 12.6, “Outliers”. OpenStax. openstax.org/books/introductory-statistics-2e/pages/12-6-out…
  17. NIST/SEMATECH. e-Handbook of Statistical Methods, section 7.1.6, “What are outliers in the data?”. itl.nist.gov/div898/handbook/prc/section1/prc16.htm
  18. Spearman, C. (1904). “The proof and measurement of association between two things”. American Journal of Psychology, 15, 72–101. Reprinted in International Journal of Epidemiology (2010), 39(5), 1137–1150. doi.org/10.1093/ije/dyq191
  19. Malinas, G. & Bigelow, J. (2004). “Simpson’s Paradox”. Stanford Encyclopedia of Philosophy (Spring 2006 Edition), ed. E. N. Zalta. plato.stanford.edu/archives/spr2006/entries/paradox-simpson
  20. Subramanian, S. V., Jones, K., Kaddour, A. & Krieger, N. (2009). “Revisiting Robinson: the perils of individualistic and ecologic fallacy”. International Journal of Epidemiology, 38(2), 342–360. pmc.ncbi.nlm.nih.gov/articles/PMC2663721
  21. Microsoft Support. “Use the Analysis ToolPak to perform complex data analysis” (the Correlation and Covariance tools). support.microsoft.com/en-us/topic/6c67ccf0-f4a9-487c-8dec-bd…
  22. Holm, S. (1979). “A simple sequentially rejective multiple test procedure”. Scandinavian Journal of Statistics, 6(2), 65–70. ime.usp.br/~abe/lista/pdf4R8xPVzCnX.pdf
  23. Benjamini, Y. & Hochberg, Y. (1995). “Controlling the false discovery rate: a practical and powerful approach to multiple testing”. Journal of the Royal Statistical Society, Series B, 57(1), 289–300. dcscience.net/Benjamini-Hochberg-1995-FDR.pdf
  24. statsmodels. “statsmodels.stats.multitest.multipletests” (documentation). statsmodels.org/stable/generated/statsmodels.stats.multitest…
  25. NIST/SEMATECH. e-Handbook of Statistical Methods, section 7.4.7.3, “Bonferroni’s method”. itl.nist.gov/div898/handbook/prc/section4/prc473.htm
  26. Penn State Eberly College of Science. STAT 505: Applied Multivariate Statistical Analysis, Lesson 6, “Multivariate Conditional Distribution and Partial Correlation” (sections 6.1–6.3). online.stat.psu.edu/stat505/book/export/html/638
  27. Cole, S. R., Platt, R. W., Schisterman, E. F., Chu, H., Westreich, D., Richardson, D. & Poole, C. (2010). “Illustrating bias due to conditioning on a collider”. International Journal of Epidemiology, 39(2), 417–420. pmc.ncbi.nlm.nih.gov/articles/PMC2846442
  28. Illowsky, B. & Dean, S. Introductory Statistics 2e, section 12.4, “Testing the Significance of the Correlation Coefficient”. OpenStax. openstax.org/books/introductory-statistics/pages/12-4-testin…
  29. Hill, A. B. (1965). “The environment and disease: association or causation?” Proceedings of the Royal Society of Medicine, 58(5), 295–300. pmc.ncbi.nlm.nih.gov/articles/PMC1898525
  30. Illowsky, B. & Dean, S. Introductory Statistics 2e, section 1.4, “Experimental Design and Ethics”. OpenStax. openstax.org/books/introductory-statistics/pages/1-4-experim…