Comparing two means

In depth

Independent samples and Welch’s t test

When the two samples are separate groups, such as hires from two bike stations or calls handled with two scripts, they are independent samples, and the question is whether the population means $\mu_1$ and $\mu_2$ differ. The null hypothesis is $H_0: \mu_1 - \mu_2 = 0$ (or a stated difference $\Delta_0$, “delta-nought”), against a one- or two-sided alternative. The estimate is the difference in sample means, and its standard error combines the two samples’ standard errors:

A two-sample t test divides the difference by this standard error. Welch’s t test compares the result with a t distribution whose degrees of freedom come from the Welch–Satterthwaite formula in the Formulas section. Its result always lies between the smaller of n₁ − 1 and n₂ − 1 at the low end and n₁ + n₂ − 2 at the high end. It is usually not a whole number, and it falls towards the low end when one sample is much noisier or much smaller than the other. The simple example runs the test on two bike-hire stations.

The pooled test, and why Welch is the safer default

The older pooled t test, Student’s two-sample t test, assumes the two populations have the same variance. It averages the two sample variances, weighted by their degrees of freedom, into a pooled variance $s_p^2$, uses $s_p\sqrt{1/n_1 + 1/n_2}$ as the standard error, and has $n_1 + n_2 - 2$ degrees of freedom. When the variances really are equal it is slightly more powerful. With equal sample sizes the two tests give the same t, and only their degrees of freedom differ.

When the variances differ and the sample sizes differ too, the pooled test can mislead. If the smaller sample has the larger variance, pooling understates the standard error and the test raises too many false alarms; if the larger sample has the larger variance, it misses real differences. The challenging example shows a pooled p-value of 0.0066 beside Welch’s 0.071 for the same calls. Delacre, Lakens and Leys (2017) show that Welch’s test controls false alarms better when variances differ and loses little when they are equal, and recommend it by default; this course follows them.

It is tempting to choose by testing the variances first. Excel’s F.TEST(array1, array2) returns the two-tailed p-value of the F test for equal variances, which is based on the ratio of the two sample variances. Using it as a gate is a poor idea: the F test is very sensitive to departures from normality (OpenStax), and Delacre and colleagues argue that choosing between the pooled and Welch tests by a preliminary test of equal variances often gives the wrong answer. Use F.TEST, if at all, to describe how different the spreads look.

T.TEST in Excel

T.TEST(array1, array2, tails, type) returns the p-value of a two-sample or paired t test, and only the p-value. Tails is 1 or 2, and type chooses the test:

Microsoft’s page says T.TEST works from a non-negative t statistic: with tails = 2 it returns the probability of a larger |t| if the two means are equal, and with tails = 1 half of that, the tail in whichever direction the data point. So a one-tailed T.TEST never exceeds 0.5 and never tells you the direction. If your one-sided alternative says μ₁ > μ₂ and the sample means say the opposite, the p-value for your alternative is 1 minus T.TEST’s: check the means before you read a one-tailed result.

A paired T.TEST (type 1) returns #N/A if the two ranges hold different numbers of values, and it pairs the values by position, so both columns must be in the same order. For the t statistic and the degrees of freedom, calculate them from the formulas or run the ToolPak.

The Analysis ToolPak’s t-tests

Data > Data Analysis offers the three tests by the names in the table above; B03 explains loading the Analysis ToolPak. Each asks for a Variable 1 Range and a Variable 2 Range, a Hypothesized Mean Difference (usually 0), whether the ranges include Labels, Alpha and an output location. The output is values, not formulas, so rerun the tool when the data change. Here is the paired tool’s output for the workshop, with the after scores as Variable 1:

Read it from the top. Pearson Correlation is the r that makes pairing pay. t Stat is $\bar{d} \div (s_d \div \sqrt{n})$ for Variable 1 minus Variable 2, so its sign depends on which range you enter first. “P(T<=t) one-tail” is, whatever its label suggests, the one-tailed p-value in the direction of t, half the two-tail value. The two critical values, 1.796 and 2.201, are positive values at the Alpha you set: reject at 5% two-sided when |t Stat| exceeds t Critical two-tail, or equivalently when P(T<=t) two-tail is below 0.05.

The equal-variances tool adds a Pooled Variance row. The unequal-variances tool calculates Welch’s degrees of freedom, and Microsoft’s page explains: “Because the result of the calculation is usually not an integer, the value of df is rounded to the nearest integer to obtain a critical value from the t table.” The tool works its p-values from that rounded df too, while T.TEST with type 3 uses the unrounded df, so the two can disagree slightly: for the calls in the challenging example, 7.83 degrees of freedom become 8 and the two-tailed p-value moves from 0.0713 to 0.0706. Neither is wrong; say which you used.

Confidence intervals for a difference

Every test here has an interval to go with it. For independent samples it is $(\bar{x}_1 - \bar{x}_2) \pm t^{*} \times \text{SE}$, with $t^{*}$ from the t distribution with Welch’s degrees of freedom; for paired samples it is $\bar{d} \pm t^{*} \times s_d \div \sqrt{n}$ with n − 1. In Excel the paired interval’s half-width is =CONFIDENCE.T(0.05,STDEV.S(D2:D13),COUNT(D2:D13)) on the column of differences, because a paired interval is a one-sample interval for the differences.

For two independent samples there is no single function: multiply =T.INV.2T(0.05,df) by the standard error. T.INV.2T truncates a fractional df to a whole number (16.69 becomes 16), which widens the interval very slightly; the simple example shows the size of the effect.

For the workshop the paired interval is 1.583 ± 2.201 × 0.4345, from 0.63 to 2.54 points: on average the trainees gained between about half a point and two and a half points. Treated wrongly as independent groups, the interval is −1.73 to 4.89 points and includes 0, matching the non-significant p of 0.33. A 95% interval for a difference excludes 0 exactly when the two-sided test at 5% rejects H₀.

Conditions

• Independence. Values within each sample must be independent. For independent samples the two groups must be separate; for paired samples the pairs must be independent of one another, so that one trainee’s change doesn’t affect another’s. Random sampling lets you generalise to a population, and random assignment to the groups lets you talk about cause.
• Normality, or large samples. The t tests assume each population is roughly normal (for a paired test, the population of differences). With larger samples the central limit theorem makes each mean’s sampling distribution close to normal even for skewed data, and how large is large enough depends on how skewed, as the sampling-distributions lesson showed with Lantern’s order values.
• No dominating outliers. A t test compares means, and one extreme value can move a mean a long way. Look at the data first, in a box plot or a histogram, and if one value drives the result, report the test with and without it.

The applied example puts these conditions to work on Lantern’s spring orders, where web and app order values are compared and one web order of $1,512 dominates.

How big is the difference?

A p-value says whether a difference is likely to be chance, not how big it is, so report an effect size with it. The plainest is the difference in means with its interval, in the units of the problem: 1.58 points out of 20, 95% interval 0.63 to 2.54. To compare effects measured on different scales, analysts also use Cohen’s d, the difference in means divided by the pooled standard deviation, $d = (\bar{x}_1 - \bar{x}_2) \div s_p$.

Cohen suggested 0.2, 0.5 and 0.8 as small, medium and large, and OpenStax stresses that these labels are only indicative; what counts as large depends on the field and the decision. For the workshop, $s_p = \sqrt{(17.90 + 12.55) \div 2} = 3.90$ and $d = 1.583 \div 3.90 \approx 0.41$: the average gain is about two-fifths of the spread between trainees. Some analysts divide a paired mean difference by $s_d$ instead, which gives 1.05 here; the two measure different things, so say which you used.

Rank tests in outline

When samples are small and skewed, or an outlier dominates, a rank test is an alternative. The Mann–Whitney U test, also called the Wilcoxon rank-sum test, replaces the values of two independent samples by their ranks in the combined data and asks whether values in one group tend to be larger than in the other. The Wilcoxon signed-rank test does the same for paired differences, using the ranks of their sizes. Neither assumes normality, and an extreme value counts only as the largest rank.

They answer a slightly different question from a t test, whether one group tends to have larger values rather than whether the means differ. Excel has no function for either, and the ToolPak has no rank test (its Rank and Percentile tool only ranks values); statistical software runs them. For right-skewed values such as order totals, a log transformation (B02’s skewed-data lesson) is another option: a t test on the logs compares geometric means.