Sunflower Intelligence
Marketing & Business Strategy

M05 · Intermediate

Market Segmentation and Positioning

10 lessons

M05 · Module 2, lesson 1 · Lesson 4 of 10

Cluster analysis

Segment customers on their needs with data: choose and standardise the variables, run k-means (and Ward’s method as a check), pick the number of segments from the elbow, the silhouette and what the segments mean, and test whether the answer is stable. Lantern’s 640-person needs study is the data.

About 95 minutes

Before you start

You will be able to

  • Choose the segmentation variables for a needs-based segmentation and explain why they are standardised first.
  • Calculate squared Euclidean distances and segment centres, run k-means by hand for two iterations and calculate the within-cluster sum of squares.
  • Explain why k-means needs k-means++ starts and restarts, why it always returns k groups, and how Ward’s hierarchical clustering and its dendrogram differ.
  • Choose the number of segments from the elbow, the mean silhouette and whether the segments mean something a business can act on, and justify the choice.
  • Assess a segmentation’s stability with restarts, the agreement between Ward’s method and k-means (the adjusted Rand index) and a split-half check.

Review first

Fictional case A fictional coffee business founded by Ana Lim. It starts as a catering and barista-training service, opens its own cafés and an online shop for beans, and plans a canned cold brew.

The idea

Fictional case

In May 2027 Ana Lim bought a needs study from a panel company for $9,600: 640 Harbour City adults who drink coffee at least once a week, recruited from an opt-in panel with quotas on age band. Each rated how much 8 needs matter when they choose coffee, from 1 (not at all important) to 7 (essential): taste and quality, a low price, convenience, sustainability, a strong caffeine kick, health (low in sugar and calories), buying from a local independent business, and new flavours. Module 1 showed that the ratings spread widely. Ana now wants to know whether the 640 fall into groups that want different things, and if so how many groups and who is in each.

OpenStax defines needs-based segmentation as dividing a market into smaller groups of people with approximately the same needs. When the groups are formed from survey answers rather than decided in advance, it is post hoc segmentation, or data-driven, as module 1 explained. The tool is cluster analysis: a family of methods that sort observations into groups, called clusters, whose members are more alike than members of different groups, judged by a measure of distance. This lesson forms the groups; Profiling segments describes, names and sizes them.

Six decisions

Every needs-based cluster analysis rests on six decisions, and two analysts given the same data can reach different segments by making them differently. Write them down before you run anything, so that they can’t be chosen to flatter the result.

  1. Segmentation variables: which answers form the segments. Here, the eight needs; who people are and what they buy come later, to describe the segments.
  2. Scale: standardise each variable, so that each counts equally in the distances.
  3. Distance: how far apart two respondents are; k-means uses squared Euclidean distance.
  4. Algorithm: k-means, started well and restarted many times; Ward’s hierarchical method as a check.
  5. Number of segments: from the elbow, the silhouette and what the segments mean.
  6. Stability: whether the segments survive restarts, another method and another half of the sample.

Respondents as points

Think of each respondent as a point with one coordinate per need: 8 coordinates for Ana’s study, two in the small examples below. After each need is standardised to a z-score (B02’s Normalisation and standardisation), one unit on every axis means one standard deviation. The squared Euclidean distance between two respondents adds up the squared differences on every need: two people who differ by one standard deviation on all 8 needs are 8×12=88 \times 1^2 = 8 apart, and two who differ by two standard deviations on price alone are 22=42^2 = 4 apart. Respondents who are close want similar things; a segment is a group of respondents close to one another.

k-means in four moves

k-means clustering sorts respondents into k groups, where k is a number you choose, so as to make the total squared distance from respondents to their group’s centre as small as possible. Each group’s segment centre (its centroid) is the mean of its members on every variable. The method repeats two moves until nothing changes:

  1. Start: choose k starting centres (k-means++ chooses them well; see In depth).
  2. Assign: put every respondent with the nearest centre, by squared Euclidean distance.
  3. Update: move each centre to the mean of the respondents assigned to it.
  4. Repeat the assign and update moves until no respondent changes group.

The total it makes small is the within-cluster sum of squares (WCSS): every respondent’s squared distance to their own centre, added up. Each assign move and each update move can only lower it or leave it unchanged, so k-means always stops. Pellmark Cycles’ eight customers, worked below, show the moves on a scale small enough to do by hand: from a poor start the WCSS falls from 82 to 21.07 after one iteration and to 8.75 after two, when one customer, G, switches group and nothing moves again.

k-means with two segments on Pellmark Cycles’ eight customers (fictional), started at customers G and H. Hollow diamonds mark the starting centres (on G and H), filled diamonds each centre after iterations 1 and 2; the final price-first centre sits on customer A and is drawn just beside it. Filled circles end in the price-first segment, open circles in the advice-first segment.

How many segments?

Ana’s analysis ran k-means on the 8 standardised needs for every k from 2 to 8, each with 20 restarts. The table shows the within-cluster sum of squares for each k, how much it fell from one k to the next, and the mean silhouette, a measure of how much closer respondents are to their own segment than to the next one (In depth defines it).

k-means on Lantern’s 640 respondents, eight standardised needs, 20 restarts for each k. With one segment the within-cluster sum of squares is the total, 5,112.
kWithin-cluster sum of squaresFall from k − 1Mean silhouetteSegment sizes
23,167.21,944.80.336328, 312
32,581.9585.30.315301, 230, 109
42,098.4483.60.291229, 154, 148, 109
51,767.9330.50.305152, 146, 129, 109, 104
61,677.790.20.280152, 146, 107, 104, 74, 57
71,597.380.40.240144, 107, 104, 86, 72, 68, 59
81,527.469.90.211107, 104, 85, 76, 74, 69, 63, 62

The within-cluster sum of squares always falls as k rises, because extra centres can only bring respondents closer to one; with 640 segments it would be 0. What matters is where the falls shrink. They are large up to k = 5 (330.5 from four to five) and then small (90.2 from five to six): the curve bends, an elbow, at five. The silhouette is highest at k = 2 (0.336), but five beats its neighbours (0.305 against 0.291 at four and 0.280 at six), a local peak.

In depth

Choosing the segmentation variables

The segmentation variables, module 1’s segmentation base, are the answers that form the segments. For a needs-based segmentation they are the needs or benefits that drive the choice the business cares about, worded so that respondents can answer them, as Ana’s eight are. Leave out who people are (age, income, region) and what they do (cups, spend, where they buy): those describe the segments afterwards, and mixing them into the clustering produces groups that are part need, part demographic and hard to interpret or reach.

Three further rules keep the variables honest. Don’t include two items that ask the same thing (“a low price” and “inexpensive”): the need they share counts twice in every distance. Leave out items that almost everyone rates alike, which add noise but no separation. And keep the number of variables in proportion to the sample: Dolnicar, Grün and Leisch’s simulations led to a guideline of at least 100 respondents per segmentation variable (an earlier study had suggested 70).

By that guideline Ana’s 8 needs call for 800 respondents, and she has 640: enough by the older figure of 560, short by the newer one. That doesn’t make the study useless, but it is one more reason to check that the segments are stable before relying on them. Ratings carry one more trap: some people rate everything high and others everything low, a habit of answering rather than a need. Standardising each variable, the next step, doesn’t remove it; analysts sometimes centre each respondent’s ratings on that respondent’s own mean, at the cost of losing whether someone cares about coffee at all. Whatever the study does, a tool that later places new customers in the segments must transform their answers in exactly the same way.

Standardising first

Distances add squared differences, so a variable with a wide spread dominates them. Standardising each variable to mean 0 and standard deviation 1 (B02) makes every need count equally. Ana’s needs share one 1–7 scale and similar spreads (standard deviations from 1.28 to 1.67), so here standardising changes little: k-means on the raw ratings gives almost the same five segments, with an adjusted Rand index (defined below) of 0.99 against the standardised ones.

It matters as soon as the scales differ. Add each respondent’s monthly coffee spend, in dollars, to the eight needs without standardising: spend’s variance (867.7) is 50 times the eight needs’ variances added together (17.2), 98.1% of the total. k-means then sorts people into spend bands ($6–32, $33–53, $54–78, $80–115 and $117–215 a month) whose agreement with the needs segments is 0.11, close to none. Its mean silhouette, 0.47, is higher than any needs solution’s, because one widely spread variable makes tidy bands: a silhouette can’t be compared across different variables or scalings. Standardised, spend becomes one variable in nine and the segments barely move (agreement 0.95), although it would still be a behaviour mixed into a needs segmentation.

Distance

k-means works with squared Euclidean distance because the mean is the point that makes the sum of squared distances to a group smallest: the update move is exactly the step that lowers the WCSS most for the current groups. Other distances exist (city-block distance adds absolute differences; others suit yes-or-no answers), and other algorithms use them, but k-means and Ward’s method as taught here are built on squared Euclidean distance. The silhouette, below, uses the ordinary Euclidean distance, the square root.

Local optima, k-means++ and restarts

k-means improves the solution it starts from, so it stops at a local optimum: a grouping that no single assign or update move can improve, though a different grouping would be better. Arthur and Vassilvitskii (2007) note that standard k-means is guaranteed only to find a local optimum, which can be quite poor. Their k-means++ start picks the first centre at random and each further centre with a probability proportional to its squared distance from the nearest centre already chosen, so the starting centres spread out. Analysts then run k-means from many such starts, called restarts, and keep the solution with the lowest WCSS.

Restarts matter on Ana’s data. Of 100 runs from a single k-means++ start each at k = 5, 60 reached the best solution (WCSS 1,767.9); the other 40 stopped at local optima between 2,015.9 and 2,201.7, 14% to 25% worse, and the worst agreed with the best solution only 0.64 on the adjusted Rand index. Even 20 restarts are a heuristic: 200 find the same solution at k = 5 but slightly lower sums at k = 6, 7 and 8 (1,677.6, 1,596.4 and 1,526.0, less than 0.1% below the table). Report how many restarts you ran.

k-means always returns k groups

To see what no structure looks like, the lesson made up 640 respondents whose 8 ratings were each drawn at random from 1 to 7, independently, and ran the same analysis with 100 restarts. k-means still returned five segments, of 121 to 138 people. Their mean silhouette was 0.095, against 0.305 for Ana’s data, and the within-cluster sum of squares fell smoothly, by 345, 251, 211, 198, 163 and 144 from one k to the next, with no elbow.

Uniform ratings don’t keep each need’s own spread, but shuffling each need’s real answers independently, which does, gives much the same silhouette (0.10). A stricter benchmark keeps the needs’ correlations with one another but builds in no clusters: normal data with Ana’s correlation matrix give 0.14 to 0.15 in three draws. Benchmarks like these, data of the same size and scale with no clusters built in, are a quick way to read what a silhouette means for your data.

Ward’s hierarchical clustering and the dendrogram

Ward’s method builds segments from the bottom up. It starts with every respondent as a group of one and, at each step, merges the two groups whose merger increases the within-cluster sum of squares least, until one group remains (Ward, 1963, proposed merging to lose least by an objective the investigator chooses; the error sum of squares is the usual one). Merging groups A and B adds nAnBnA+nB\frac{n_A n_B}{n_A + n_B} times the squared distance between their centres to the WCSS, so small groups with close centres merge first. The record of merges is drawn as a dendrogram, a tree whose branches join at the height of each merge; cutting it at any height gives that many segments.

Ward’s method on Pellmark Cycles’ eight customers (fictional). Each pair joins at the merge height SciPy and the site’s engine use for Ward’s method, √(2 × the merge’s cost): A and C, whose merge costs 0.50 by the card below, join at height 1, and the last merge, at 10.97, costs 10.97² ÷ 2 = 60.13, exactly the fall from one segment to two in the worked example “Two segments or three?”. It costs far more than any merge before it, which is why a cut below it gives two segments.

Ward’s method needs no k in advance and shows every number of segments at once, but it never undoes a merge, so an early merge that later looks wrong stays, and it compares every pair of groups, which grows slow for tens of thousands of respondents. k-means needs k and a good start but can move respondents freely. The two are often used together: Ward’s tree to see which values of k look natural, k-means to refine the groups. On Ana’s data, Ward’s five groups and the k-means segments agree closely: 624 of 640 respondents, 97.5%, are in matching groups, and 16 differ.

Comparing two segmentations: the adjusted Rand index

Two segmentations of the same people can label their segments differently, so they are compared over pairs of respondents: does each pair sit together in both, or apart in both? The adjusted Rand index of Hubert and Arabie (1985) counts that agreement and corrects it for chance: 1 means identical groupings (whatever the labels), about 0 means no more agreement than random labels would give, and it can be negative. Ward against k-means on Ana’s data scores 0.93.

Choosing k: the elbow, the silhouette and meaning

The elbow method plots the within-cluster sum of squares against k and looks for the bend after which extra segments buy little. The bend is often soft, so read the falls, not only the picture. With standardised data the total sum of squares is (n−1)p=639×8=5,112(n - 1)p = 639 \times 8 = 5,112, so Ana’s five segments leave 65.4% of the variation in the needs between segments and 34.6% within them.

The silhouette of a respondent, proposed by Rousseeuw (1987), compares a, their average distance to the other members of their own segment, with b, their average distance to the members of the nearest other segment: s = (b − a) ÷ max(a, b). It runs from −1 to 1. Near 1, the respondent sits well inside their segment; near 0, between two segments; below 0, nearer another segment than their own. The mean silhouette over all respondents summarises the solution, and Rousseeuw suggested it might help select the number of clusters. A widely quoted rule of thumb from Kaufman and Rousseeuw’s book on cluster analysis (1990, as reproduced by Emamat and colleagues) reads a mean of 0.71 or more as strong structure, 0.51 to 0.70 as reasonable structure, 0.26 to 0.50 as weak structure that could be artificial, and 0.25 or less as no substantial structure.

Read the size of a silhouette, not only its peak. Ana’s 0.305 at k = 5 is about three times the 0.095 of random ratings and about twice the 0.14–0.15 of correlated data without clusters, so the segments are not an accident of the method; on Kaufman and Rousseeuw’s scale it is weak structure, far from the values near 1 of tight, well-separated groups. The segments overlap, and many respondents sit near a boundary. The silhouette often peaks at a small k when one split dominates the data, as here: at k = 2 it is highest, but the two groups are 312 people who put price (index 120 against the average respondent), convenience (119) and caffeine (118) first, and 328 who rate taste, sustainability, health, local and novelty above average: true, and too coarse to plan five different offers around.

Meaning settles it. At k = 4, most of segment D (local and taste first) and most of segment C (novelty first) merge into one group of 229, although a café that sells itself on Harbour City roots and a brand that launches new flavours every month are different propositions. At k = 6 (the best of 200 restarts), segment C divides between a group of 76 (74 of C’s members and 2 from D; novelty index 165, with taste and local above average) and one of 55 (novelty 148, every other need below average): the same need at two strengths, not a new one. Dolnicar, Grün and Leisch put it plainly: statistical criteria are no substitute for the judgement of the people who will use the segments.

Stability

A segmentation worth acting on should not depend on the luck of one run. Three checks are quick: whether many restarts reach the same best solution (60 of 100 single starts did), whether a different method finds similar groups (Ward against k-means: an adjusted Rand index of 0.93), and whether another half of the sample gives the same segments. For the last, Ana’s analyst segmented the first 320 respondents and the last 320 separately, assigned all 640 to each half’s nearest centres, and compared the two assignments: an adjusted Rand index of 0.97.

Dolnicar and Leisch go further, re-running the segmentation on many bootstrap samples (samples of the same size drawn with replacement) and measuring how often each segment reappears. They distinguish natural segmentation, when the data hold real clusters; reproducible segmentation, when the data have structure that gives the same solution again and again without clear clusters; and constructive segmentation, when the data have no structure and the analyst and the business choose among many artificial solutions. Ana’s segments look reproducible: stable across starts, methods and halves, but overlapping. Profiling segments adds a fourth check, assigning held-back respondents to the segments.

Formulas

Squared Euclidean distance between two respondents

d2(zi,zh)=∑j=1p(zij−zhj)2d^2(\mathbf{z}_i, \mathbf{z}_h) = \sum_{j=1}^{p} (z_{ij} - z_{hj})^2
d2(zi,zh)d^2(\mathbf{z}_i, \mathbf{z}_h)
Squared distance between respondents (or centres) i and hsquared standard deviations
zij, zhjz_{ij},\ z_{hj}
Their standardised ratings (z-scores) on segmentation variable jstandard deviations
pp
Number of segmentation variables

Assumes. Every variable is standardised first, so each counts equally.

B02’s Euclidean distance card is the square root of this; its standardisation card gives the z-scores. In Excel, with two rows of z-scores in B2:I2 and B16:I16: =SUMPRODUCT((B2:I2-B16:I16)^2).

Segment centre (centroid)

cgj=1ng∑i∈gzijc_{gj} = \dfrac{1}{n_g} \sum_{i \in g} z_{ij}
cgjc_{gj}
Centre of segment g on variable j: its members’ meanstandard deviations
ngn_g
Number of respondents in segment grespondents
zijz_{ij}
Standardised rating of respondent i on variable jstandard deviations

The update move of k-means recomputes every centre this way. In Excel, with the standardised ratings (z-scores) in B2:I641 and each respondent’s segment in J2:J641, =AVERAGEIFS(B$2:B$641,$J$2:$J$641,$L2) gives the centre of the segment named in L2 on the variable in column B.

Within-cluster sum of squares (WCSS)

Wk=∑g=1k∑i∈gd2(zi,cg),T=(n−1) pW_k = \sum_{g=1}^{k} \sum_{i \in g} d^2(\mathbf{z}_i, \mathbf{c}_g), \qquad T = (n - 1)\,p
WkW_k
Within-cluster sum of squares of a k-segment solutionsquared standard deviations
d2(zi,cg)d^2(\mathbf{z}_i, \mathbf{c}_g)
Squared distance from respondent i to the centre of their own segment gsquared standard deviations
kk
Number of segments
TT
Total sum of squares: W with one segment; (n − 1)p for p standardised variablessquared standard deviations
nn
Number of respondentsrespondents
pp
Number of segmentation variables

Assumes. At its best, W falls (or stays level) as k rises, and is 0 when every respondent is a segment of one; k-means may stop above the best.

1 − W ÷ T is the share of the variation that lies between segments. scikit-learn calls W the inertia.

Silhouette of a respondent, and the mean silhouette

s(i)=b(i)−a(i)max⁡{a(i), b(i)},sˉ=1n∑i=1ns(i)s(i) = \dfrac{b(i) - a(i)}{\max\{a(i),\ b(i)\}}, \qquad \bar{s} = \dfrac{1}{n} \sum_{i=1}^{n} s(i)
a(i)a(i)
Average Euclidean distance from respondent i to the other members of their own segmentstandard deviations
b(i)b(i)
Lowest average distance from i to the members of another segment (the nearest other segment)standard deviations
s(i)s(i)
Silhouette of respondent i, from −1 to 1
sˉ\bar{s}
Mean silhouette of the solution
nn
Number of respondentsrespondents

Assumes. A respondent alone in a segment gets s = 0 by convention.

Rousseeuw (1987). Compare the mean silhouette only between solutions on the same variables and scaling.

Ward’s merge cost

Δ(A,B)=nA nBnA+nB d2(cA,cB)\Delta(A, B) = \dfrac{n_A\, n_B}{n_A + n_B}\, d^2(\mathbf{c}_A, \mathbf{c}_B)
Δ(A,B)\Delta(A, B)
Increase in the within-cluster sum of squares if groups A and B mergesquared standard deviations
nA, nBn_A,\ n_B
Numbers of respondents in A and Brespondents
d2(cA,cB)d^2(\mathbf{c}_A, \mathbf{c}_B)
Squared distance between their centressquared standard deviations

Assumes. Squared Euclidean distances on standardised variables.

At each step Ward’s method merges the pair with the smallest Δ. Two respondents alone merge at half their squared distance.

Worked examples

A first example: k-means by hand at a bike shop

Fictional figures

Pellmark Cycles asked eight customers how much two things matter when they buy, from 1 (not at all) to 7 (essential): a low price and expert advice. Run k-means with k = 2, starting from customers G and H as the two centres, until no customer changes segment.

Pellmark Cycles: eight customers’ ratings (fictional)
CustomerLow priceExpert advice
A62
B71
C52
D63
E26
F17
G35
H25
  1. Check the scale. Both needs use the same 1–7 scale, with standard deviations of 2.27 and 2.17; standardising them gives exactly the same two segments (an adjusted Rand index of 1), so this example works in rating points to keep the arithmetic readable.
  2. Assign (iteration 1). Find each customer’s squared distance to G (3, 5) and to H (2, 5). A: (6 − 3)² + (2 − 5)² = 18 to G and (6 − 2)² + (2 − 5)² = 25 to H, so A joins G. B (32 against 41), C (13 against 18) and D (13 against 20) join G too; E (2 against 1) and F (8 against 5) join H. Around the starting centres the WCSS is 82.
  3. Update. The centre of G’s group (A, B, C, D, G) is its members’ mean: low price (6 + 7 + 5 + 6 + 3) ÷ 5 = 5.4, advice (2 + 1 + 2 + 3 + 5) ÷ 5 = 2.6. H’s group (E, F, H) moves to (2 + 1 + 2) ÷ 3 = 1.67 and (6 + 7 + 5) ÷ 3 = 6.00. The WCSS falls to 21.07.
  4. Assign and update (iteration 2). G is now 11.52 from (5.4, 2.6) and 2.78 from (1.67, 6.00), so G switches; no one else moves. The centres become (6, 2) for A, B, C and D and (2, 5.75) for E, F, G and H, and the WCSS falls to 8.75.
  5. Check. A third assignment moves no one: G, the closest call, is 18 from (6, 2) and 1.56 from (2, 5.75). k-means stops.

Answer. Two segments: A, B, C and D, who put price first (centre: price 6, advice 2), and E, F, G and H, who put advice first (centre: price 2, advice 5.75). Within-cluster sum of squares 8.75.

What it means. The poor start, two neighbouring advice-seekers, cost an extra iteration here and nothing more. With hundreds of respondents and eight needs, a poor start can leave k-means at a worse answer for good, which is why real analyses use k-means++ starts and many restarts.

A harder example: Two segments or three?

Fictional figures

Pellmark’s owner asks whether three segments would describe the eight customers better than two. Compare the solutions for k = 1 to 4 on the within-cluster sum of squares and the mean silhouette, work out customer G’s silhouette in the two-segment solution, and decide.

Pellmark Cycles: k-means solutions (best of 50 starts)
kWithin-cluster sum of squaresMean silhouetteSegment sizes
168.88—8
28.750.714, 4
35.330.504, 3, 1
42.670.313, 3, 1, 1
  1. Read the within-cluster sum of squares. It falls at every k, from 68.88 (one segment: the total) to 8.75, 5.33 and 2.67, and would reach 0 at k = 8. The first fall (60.13) dwarfs the next (3.42): the elbow is at two.
  2. Work out G’s silhouette for k = 2, with ordinary (not squared) distances. Within G’s segment: to E √2, to F √8 and to H 1, so a = (1.414 + 2.828 + 1) ÷ 3 = 1.748. To the other segment: to A √18, to B √32, to C √13 and to D √13, so b = (4.243 + 5.657 + 3.606 + 3.606) ÷ 4 = 4.278. s = (4.278 − 1.748) ÷ 4.278 = 0.59.
  3. Compare the mean silhouettes, each the average of all eight customers’ silhouettes worked the same way: 0.71 for two segments, 0.50 for three and 0.31 for four. Three segments separate the customers less well than two.
  4. Look at what the third segment is. k-means with k = 3 puts customer F, the keenest advice-seeker, in a segment of one (whose silhouette is 0 by convention). A segment of one customer isn’t substantial, one of module 1’s tests of a usable segment, and couldn’t justify an offer of its own.

Answer. Two segments. The lower within-cluster sum of squares at k = 3 (5.33 against 8.75) is guaranteed and proves nothing; the mean silhouette falls from 0.71 to 0.50, and the third segment is one customer.

What it means. Eight customers make the comparison unusually clear. In a real survey the elbow is softer, the silhouette smaller and the segments overlap, so the decision leans on what the segments mean, as Lantern’s does.

Applied: a business decision: Choosing and checking Lantern’s segments

Fictional figures

Ana’s analyst has run k-means on the 640 respondents’ eight standardised needs for k = 2 to 8 (the table in The idea). Ana must decide how many segments to take into profiling, and whether she can trust them.

  1. Elbow: the within-cluster sum of squares falls by 1,944.8, 585.3, 483.6 and 330.5 up to k = 5, then by only 90.2, 80.4 and 69.9. The bend is at five, where the segments leave 65.4% of the variation between them.
  2. Silhouette: highest at two (0.336), a local peak at five (0.305, against 0.291 and 0.280). Two is a broad split, price and practicality against quality and values; five is the candidate.
  3. Meaning: at five, the segments’ highest indexes against the average respondent are price (142) for A, energy (147) and convenience (137) for B, novelty (158) for C, local (168) and taste (136) for D, and health (168) and sustainability (159) for E. Each calls for a different offer. Four would merge C and D; six would split C into two strengths of the same need.
  4. Stability: 60 of 100 single starts reach the same best solution; Ward’s method agrees (adjusted Rand index 0.93; 16 of 640 respondents placed differently); the two halves of the sample give the same segments (0.97).
  5. Caveats to report: the structure is real but weak (a mean silhouette of 0.305, against 0.095 for random ratings), the sample is below the guideline of 800 for eight variables, and it comes from an opt-in panel, so the segments describe these 640 people first.

Answer. Five segments, of 152, 146, 129, 109, 104 respondents (lettered A to E by size), stable across restarts, Ward’s method and the two halves of the sample.

What it means. Ana can take five segments into profiling with confidence that they are not an accident of one run, while saying plainly that they are a model with overlapping edges, not five separate populations. Their names, sizes in spend and the typing tool that assigns new customers come next.

Explore

All 640 respondents’ eight needs. Start at k = 5 and read the profile table: which needs does each segment rate above 100? Then try k = 4 and k = 6 and see which segments merge and split. Untick needs to see how the segments change when a need is left out, and untick “Standardise” to see how little it matters for ratings on one scale. The widget runs 12 restarts, so at k = 7 it may stop at a higher sum (1,607.6) than the table’s 1,597.3, which is itself a local optimum: 200 restarts reach 1,596.4. At k = 6 its split of segment C may also differ a little from the one described in In depth.

Loading the interactive…

The same needs with monthly coffee spend in dollars added as a ninth variable. Untick “Standardise” and look at the segments’ spend per head: they become spend bands, with a silhouette (0.47 at k = 5) higher than any needs solution’s. Tick it again and the needs take over. Then untick “Monthly spend ($)” to return to the needs alone.

Loading the interactive…

Common mistakes

  • “The clusters are the segments that exist in the market.”

    k-means returns k groups from any data: 640 made-up respondents with random ratings still gave five segments. Clusters are a model you choose and check. Report how clear the structure is (the silhouette against a benchmark) and how stable it is, and call the segments what they are: groups of people with similar answers, with overlapping edges.

  • “Choose the k with the lowest within-cluster sum of squares.”

    The WCSS falls every time k rises and reaches 0 when every respondent is a segment of one. Look for where the falls shrink (the elbow), not for the lowest value.

  • “The highest silhouette decides the number of segments.”

    The silhouette often peaks at a small k when one split dominates the data: Ana’s peaks at two, a split too coarse to plan offers around. It can’t be compared across different variables or scalings either (raw spend bands score 0.47). Use it with the elbow, stability and what the segments mean.

  • “One run of k-means gives the answer.”

    k-means stops at a local optimum that depends on its start: 40 of 100 single starts on Ana’s data stopped at worse solutions. Use k-means++ starts, run many restarts, keep the lowest WCSS and report how many you ran.

  • “Put every variable in the survey into the clustering.”

    Demographics and behaviour mixed with needs give segments that are part need, part demographic, and hard to interpret. Duplicated items double-count a need, and every extra variable needs more respondents (about 100 each, by Dolnicar, Grün and Leisch’s guideline). Segment on the needs; describe with the rest.

  • “Standardising is a formality.”

    It decides what the distances measure. On Ana’s 1–7 ratings it changes little, but add spend in dollars unstandardised and spend is 98% of the variance: the segments become spend bands that agree with the needs segments only 0.11.

Guided practice

Work through these with hints when you need them. Every hint and every worked solution is there to be used.

Loading four questions…

On your own

No hints this time. Answer, check, and read the feedback; the solution appears once you have answered or if you ask for it. Your first answers count toward mastery.

Loading six questions…

In practice

Fictional case

Tollwick Software, a fictional firm that sells scheduling software to small businesses, wants to rebuild its onboarding around what different customers need. Its customer-insight lead, Teodora Rafferty, surveyed 1,150 business customers on 7 needs, from easy set-up to integration with accounting tools, each rated 1–7.

She writes the specification first. Before looking at any results, Teodora records the segmentation variables (the 7 needs, with firm size, sector and plan kept for describing), standardisation, k from 2 to 7, k-means with k-means++ starts and 50 restarts, Ward’s method as a check, and the stability tests. With 1,150 customers she is above the guideline of 100 a variable (700).

She shows the team three solutions, not one. The silhouette is highest at three segments (0.31), with four close behind (0.29) and five lower (0.24). At four, the largest of the three segments splits into customers who want integrations and customers who want reports, and the product team points out that those two need different onboarding paths. Her split-half check gives an adjusted Rand index of 0.91 for four segments and 0.62 for five, so five is not stable enough to plan around.

She reports the choice with its evidence and its limits. The team adopts four segments (362, 301, 274, 213 customers), and her one-page summary gives the specification, the elbow and silhouette for every k, the stability results and a sentence on what the segments are not: “four groups of customers with similar needs, overlapping at the edges, not four kinds of customer with nothing in common.” She plans to re-run the segmentation each year with the same specification, so that a change in the segments reflects a change in customers rather than in the method.

Ana’s analysis followed the same pattern on a smaller scale, and her next job is the one Teodora’s team faced after choosing: profile each segment, give it an honest name, size it in people and in spend, and build a short questionnaire that assigns new customers to a segment.

Knowledge check

Six questions, one attempt each, no hints. Score 80% or more to complete the lesson; you can take it again at any time.

Loading six questions…

Summary

  • Needs-based (post hoc) segmentation forms segments from customers’ needs with cluster analysis; demographics and behaviour describe the segments afterwards.
  • Choose a few relevant, non-duplicated needs, with about 100 respondents per variable, and standardise them so each counts equally in the distances.
  • k-means assigns each respondent to the nearest centre and moves each centre to its members’ mean until nothing changes, lowering the within-cluster sum of squares at every step.
  • k-means stops at local optima, so use k-means++ starts and many restarts, and remember it returns k groups even from data with no structure.
  • Ward’s method merges the two groups that raise the WCSS least, shows every k in a dendrogram, and is a useful check on k-means.
  • Choose k from the elbow, the mean silhouette (s = (b − a) ÷ max(a, b)) and whether the segments mean something the business can act on; Lantern’s data give five.
  • Check stability with restarts, a second method and a split-half test, comparing groupings with the adjusted Rand index (1 identical, about 0 chance).

Key formulas

Terms introduced

Cluster analysis
A family of methods that sort observations into groups (clusters) whose members are more alike than members of different groups, judged by a chosen measure of distance. In marketing it forms segments from survey answers.
k-means clustering
A cluster analysis that sorts observations into k groups by repeatedly assigning each to the nearest group centre and moving each centre to the mean of its members, until no observation changes group; it makes the within-cluster sum of squares small.
Segment centre
The mean of a segment’s members on every segmentation variable; also called the centroid. In k-means each observation belongs to the segment with the nearest centre.
Squared Euclidean distance
The sum of the squared differences between two observations on every variable, Σ(x_j − y_j)²; the square of the straight-line distance.
Within-cluster sum of squares
The sum of every observation’s squared distance to the centre of its own cluster (WCSS); k-means makes it small, and it always falls as the number of clusters rises.
Local optimum
A solution that no small step can improve although a better solution exists elsewhere; k-means stops at one that depends on its starting centres.
k-means++
A way of choosing k-means’ starting centres: the first at random, each further one with probability proportional to its squared distance from the nearest centre already chosen, so the starts spread out (Arthur and Vassilvitskii, 2007).
Elbow
The point on a plot of the within-cluster sum of squares against the number of clusters after which extra clusters reduce it only a little; one guide to the number of clusters.
Silhouette
For one observation, (b − a) ÷ max(a, b), where a is its average distance to the other members of its cluster and b its average distance to the nearest other cluster; from −1 to 1, near 1 when it sits well inside its cluster (Rousseeuw, 1987). The mean silhouette summarises a solution.
Ward’s method
Hierarchical clustering that starts with every observation alone and repeatedly merges the two groups whose merger increases the within-cluster sum of squares least.
Dendrogram
A tree diagram of a hierarchical clustering, in which branches join at the height of each merge; cutting it at a height gives that many clusters.
Adjusted Rand index
A measure of agreement between two groupings of the same observations, counted over pairs and corrected for chance: 1 for identical groupings, about 0 for chance agreement (Hubert and Arabie, 1985).

Sources

  1. Gomez Albrecht, M., Green, M. & Hoffman, L. (2023). Principles of Marketing, chapter 5, “Market Segmentation, Targeting, and Positioning”: key terms (needs-based segmentation; market segmentation). OpenStax. openstax.org/books/principles-marketing/pages/5-key-terms
  2. Dolnicar, S., Grün, B. & Leisch, F. (2018). Market Segmentation Analysis: Understanding It, Doing It, and Making It Useful. Springer (Management for Professionals), open access under CC BY 4.0. Chapter 2 (natural, reproducible and constructive segmentation), Step 5, “Extracting segments”, and Step 6, “Profiling segments” (statistical criteria are no substitute for user assessment). link.springer.com/book/10.1007%2F978-981-10-8818-6
  3. Dolnicar, S. (2021). Data-driven market segmentation analysis. In Women’s Voices in Tourism Research, chapter 37. The University of Queensland. (States the guideline of 100 respondents per segmentation variable from Dolnicar, Grün & Leisch, 2016, and the earlier 70; summarises natural, reproducible and constructive segmentation.) uq.pressbooks.pub/tourismknowledge/chapter/market-segmentati…
  4. Dolnicar, S., Grün, B. & Leisch, F. (2016). Increasing sample size compensates for data problems in segmentation studies. Journal of Business Research, 69(2), 992–999. doi:10.1016/j.jbusres.2015.09.004. (The simulations behind the 100-per-variable guideline; the figure itself is given in Dolnicar, 2021, not in this abstract.) ideas.repec.org/a/eee/jbrese/v69y2016i2p992-999.html
  5. Dolnicar, S. & Leisch, F. (2010). Evaluation of structure and reproducibility of cluster solutions using the bootstrap. Marketing Letters, 21(1), 83–101. doi:10.1007/s11002-009-9083-4. ideas.repec.org/a/kap/mktlet/v21y2010i1p83-101.html
  6. Arthur, D. & Vassilvitskii, S. (2007). k-means++: the advantages of careful seeding. Proceedings of the 18th Annual ACM-SIAM Symposium on Discrete Algorithms, 1027–1035. haralick.org/ML/kMeansPP-soda.pdf
  7. Ward, J. H., Jr. (1963). Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58, 236–244. doi:10.1080/01621459.1963.10500845. doi.org/10.1080/01621459.1963.10500845
  8. Rousseeuw, P. J. (1987). Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20, 53–65. doi:10.1016/0377-0427(87)90125-7. doi.org/10.1016/0377-0427(87)90125-7
  9. Kaufman, L. & Rousseeuw, P. J. (1990). Finding Groups in Data: An Introduction to Cluster Analysis. Wiley. Their rough scale for the mean silhouette as reproduced in Emamat, M. S. M. M. et al. (2022), Using ELECTRE-TRI and FlowSort methods in a stock portfolio selection context, Financial Innovation, 8(1), table 16 (the book itself was not consulted). jfin-swufe.springeropen.com/articles/10.1186/s40854-021-0031…
  10. Hubert, L. & Arabie, P. (1985). Comparing partitions. Journal of Classification, 2, 193–218; the adjusted Rand index as documented in scikit-learn 1.9 (adjusted_rand_score). scikit-learn.org/stable/modules/generated/sklearn.metrics.ad…