The idea
k-means places k centres and assigns each row to the nearest, moving the centres until the within-group squared distance is as small as it gets. You choose k. Ward’s hierarchical clustering starts with every row alone and keeps merging the two groups whose merge adds least spread. The dendrogram draws every merge at its height: cut it lower for more, smaller groups; higher for fewer. Standardise the columns first in both.
Two methods rarely agree exactly. The adjusted Rand index measures how alike two groupings are: 1 means identical, about 0 means no more alike than chance. Low agreement often means there is little real structure to find, not that one method is wrong.
Principal component analysis (PCA) finds new axes, each a weighted mix of the standardised columns, ordered by how much variation they carry. Each one’s share is its eigenvalue divided by the number of columns. In a biplot, long arrows pointing the same way mean positively correlated columns, and opposite ways negatively correlated.
The formula
Ward merge cost Δ(A, B) = nA nBnA + nB ‖cA − cB‖2, share of PCk = λkp
- nA, cA
- the size and centre (mean) of group A; Ward merges the pair with the smallest cost
- λk
- the k-th largest eigenvalue of the correlation matrix
- p
- the number of columns (the eigenvalues add up to p)
- score
- a row’s position on component k: Σ wjkzj, each column’s weight times the row’s standardised value
The dendrogram draws each merge at √(2Δ), the same heights as SciPy.
A worked example
Customers’ annual spend and store visits, both standardised, correlate at 0.8. Run PCA on the two columns.
- The correlation matrix is [1, 0.8; 0.8, 1]. For two columns its eigenvalues are 1 + r = 1.8 and 1 − r = 0.2; they add up to 2, the number of columns.
- The first component carries 1.8 ÷ 2 = 90% of the variation; the second, 10%.
- Its weights are 0.707 on each column: an equal mix, best read as overall engagement. The second component’s weights are 0.707 and −0.707: spend relative to visits.
- A customer 1.2 standard deviations above average on spend and 0.8 on visits scores 0.707 × 1.2 + 0.707 × 0.8 = 1.41 on the first component and 0.707 × 1.2 − 0.707 × 0.8 = 0.28 on the second.
Two columns become one engagement score that keeps 90% of their spread. Each column correlates 0.95 with it, which is why their biplot arrows are long and point the same way.
Try it
How many directions do three columns need?
- Second component
- 21%
- First two together
- 90%
- First component’s weights
- +0.62, +0.59, +0.51
Two directions carry almost everything: a two-dimensional map of the rows loses little.
The common mistake
Running PCA or clustering on unscaled columns
Take income in dollars (standard deviation $20,000) and age in years (standard deviation 12), correlated at 0.3. Without standardising, the first component carries over 99.99% of the variance and is income alone, because its numbers are bigger. Standardised, the first component carries 65% and weighs both columns equally. Distances in k-means and Ward suffer the same way.
Remember
- Cutting a dendrogram lower gives more, smaller groups. Standardise before any distance-based method.
- Adjusted Rand: 1 is identical groupings, about 0 is chance. Low agreement can mean there is no strong structure.
- A component is a weighted mix of all the columns; its share of variance is its eigenvalue ÷ the number of columns.
Check yourself
Three questions from the Learn room’s bank. Your answers update your mastery of this skill, the same as in Practise.
Use it on real numbers
Open Principal components, in Business analytics › Unsupervised learning and try the idea on your own numbers.