Describing distributions

In depth

Why the sample variance divides by n − 1

Take a population of just three values, 2, 4 and 9 (an illustration, not Lantern data). Its mean is $\mu = 5$ and its variance, dividing by $N = 3$, is $\sigma^2 = (9 + 1 + 16) \div 3 \approx 8.67$. Draw a sample of two values at random, one after the other, putting the first back before the second is drawn. There are nine equally likely samples. For each, the table calculates the variance both ways.

Averaged over all possible samples, dividing by $n - 1$ gives $8.67$, exactly the population variance; dividing by $n$ gives $4.33$, only half of it. In general the divide-by-$n$ version averages $(n - 1) \div n$ of $\sigma^2$: a large error for tiny samples and a negligible one for large samples, which is why STDEV.S and STDEV.P give $100.90 and $100.68 for the 231 orders but differ by 4.7% for 11 launch events. An estimate that is right on average is called unbiased, so $s^2$ is an unbiased estimate of $\sigma^2$. (The square root spoils this slightly: $s$ is still a little low on average, but much less so than the divide-by-$n$ version.)

Another way to say it uses degrees of freedom. The $n$ deviations from the sample mean must add to 0, so once you know $n - 1$ of them the last is fixed: only $n - 1$ are free to vary. The sample variance averages the squared deviations over those $n - 1$ degrees of freedom. The same idea returns in the t, chi-square and F distributions later in the course.

Adding a constant and changing units

Suppose every value is transformed as $y = a + bx$: $a$ is added and $b$ multiplies. Measures of centre and position follow the transformation: $\bar{y} = a + b\bar{x}$, and the median and quartiles move the same way. Measures of spread ignore $a$ and scale by $|b|$: $s_y = |b| s_x$, and the interquartile range (IQR) likewise, while the variance scales by $b^2$. z-scores, skewness and kurtosis don’t change at all when $b$ is positive, because they are measured in standard deviations.

Two Lantern illustrations. If Ana had added a flat $5.20 delivery charge to every spring order (a hypothetical), the mean would rise from $39.38 to $44.58 and the median from $19.50 to $24.70, but the standard deviation and the IQR would stay at $100.90 and $23.60. And the 40 Kilnworth bags she weighed in October average 1,002.75 g with a standard deviation of 2.66 g: in kilograms ($b = 1 \div 1{,}000$) that is 1.00275 kg and 0.00266 kg, and as overfill above the label ($y = x - 1{,}000$) it is 2.75 g and still 2.66 g.

The coefficient of variation

A standard deviation is in the data’s units, so it can’t compare the spread of groups with very different means, or of variables in different units. The coefficient of variation (CV) divides the standard deviation by the mean: $\text{CV} = s \div \bar{x}$, usually written as a percentage. It measures spread relative to the size of the values. For Ana’s catering, launches vary by 7.5% of their mean per guest-day and meetings by 22.3%: meetings are three times as variable, relative to what they charge.

The CV only makes sense for a variable measured from a true zero (a ratio scale, in B01’s terms) with a positive mean, such as money, weight or time. Kilnworth’s bag weights have a CV of 0.26%: they vary little relative to their size. The same bags’ overfill above 1,000 g has a CV of 97%, although nothing about the bags has changed. Overfill has no true zero and can be negative (5 of the 40 bags were under 1,000 g); moving the zero changed the mean but not the standard deviation, so the ratio means little. Don’t compute a CV for temperatures in °C, z-scores, profits that can be negative, or ratings.

Percentiles and percentile ranks

A percentile (B03) goes from a percentage to a value: the 90th percentile of order values is the value with 90% of orders at or below it. A percentile rank goes the other way, from a value to a percentage. OpenStax’s Introductory Business Statistics uses $(B + 0.5E) \div n \times 100$, where $B$ is the number of values below the one you are ranking and $E$ the number equal to it. Excel’s =PERCENTRANK.INC(array,x) returns a fraction instead: for a value in the data, it is the number of values below it divided by $n - 1$. Without ties, that is the share of the other values that lie below it.

Excel cuts PERCENTRANK.INC off at three digits unless you ask for more, rather than rounding: Microsoft’s own example returns 0.555 for 5 ÷ 9. Counting gives the same fraction without the truncation: =COUNTIF(B2:B29,"<"&E2)/(COUNT(B2:B29)-1). For a value that isn’t in the data, PERCENTRANK.INC interpolates between its neighbours, which counting doesn’t. Microsoft’s page doesn’t say how ties are treated; the worksheets on this site count only the values strictly below, as in the definition above.

With many ties the two definitions of a percentile rank can disagree badly. 35 of the 154 star ratings are below 4 and 51 equal 4, so a 4-star rating sits at the 39th percentile by OpenStax’s formula and at 0.228 by PERCENTRANK.INC (35 ÷ 153 = 0.2288, cut to three digits). State the method whenever ties are common.

The five-number summary and box plots

The five-number summary is the minimum, the first quartile, the median, the third quartile and the maximum. It shows the centre, the spread of the middle half and the reach of both tails, and it is what a box plot draws, as B03’s box and whisker chart did. The table and the figure compare Ana’s five kinds of catering event by invoice per guest-day (a two-day event counts its guests on both days).

Read the picture before the table. Team days charge most per guest (median $11.19) and launches least and most consistently (median $9.08, the narrowest box). Meetings have a modest median, $9.22, but a long right whisker and two outliers: their mean, $9.63, sits well above it. Every outlier lies on the high side, and the next section explains why.

Skewness and kurtosis

Excel’s SKEW measures skewness. It cubes each value’s z-score, so values far out on one side dominate it: positive for a long right tail, negative for a long left one, near 0 for symmetric data (B02, B03). The meetings’ SKEW is 2.05, and their mean is above their median, the usual sign of a right tail. The tail has a business cause: invoices include a fixed set-up charge and a minimum, so the smallest events pay most per guest. EV-047, the meeting at $17.14 per guest, had 35 guests, the fewest of any meeting.

KURT, Excel’s excess kurtosis, uses fourth powers and is about 0 for normally distributed data. Microsoft’s page describes it as the “relative peakedness or flatness” of a distribution, but Westfall (2014) showed that kurtosis says almost nothing about the peak: it measures how extreme the tails are, that is, how prone the data are to outliers. Lantern’s orders have a KURT of 199.5 because of one order, not because of a sharp peak. Read a large KURT as “check the extremes”.

Describing a categorical variable

For a categorical variable the summaries are counts, proportions and the mode. Of the 231 spring orders, 132 came through the web shop and 99 through the app, proportions of 0.571 and 0.429; the mode is web. A 0/1 variable is a special case: the mean of Lantern’s subscription flags, 1 for a subscription order and 0 otherwise, is 0.450, the proportion of subscription orders (104 of 231). That is why later lessons treat a proportion as the mean of a 0/1 variable.

Ratings are ordinal, so the median and percentiles are meaningful while the mean, 4.10, is a convenience that assumes equal steps between stars (B01). The mode is 5 stars, and the median is 4: the 77th and 78th of the 154 sorted ratings both fall among the 4s, which run from position 36 to 86. The table tells more than any one number: 77% of rated orders got 4 or 5 stars, and 8% got 1 or 2.

Comparing groups side by side

• Use the same summaries, the same quartile method and the same scale for every group, and show each group’s size: a mean of 13 events and a mean of 28 don’t carry the same weight.
• Match the summary to the shape. For roughly symmetric groups the mean and standard deviation describe them; for skewed groups, or groups with outliers, add the median and IQR.
• Use relative measures to compare spread across groups of different size or units: the CV, the IQR as a share of the median for skewed groups, or z-scores within each group.
• Describing a difference is not testing it. Whether team days really earn more per guest than launches, or only did in these 76 events, is the question of ANOVA in module 4.