Hypothesis testing

In depth

Writing the hypotheses

The null hypothesis is the claim being checked, or the “no change, no difference” position: the specification, the target, last year’s average. It always contains the equality (=, ≤ or ≥), because the test needs one exact value to work out how the statistic behaves. For H₀: μ ≥ 1,004 g the test uses the boundary, 1,004 g, the case hardest to reject. The alternative is what the data would have to show, and it contains <, > or ≠.

Choose one- or two-sided from the decision, before seeing the data. A one-sided test fits when only one direction would change what you do and the other would be treated like no difference: Ana acts on underfilling, while slightly heavy bags cost Kilnworth, not her. When either direction matters, use a two-sided test: a cleaning time that rose would break a hotel’s rota, and one that fell would free staff. Picking the side after seeing which way the data went, or switching to one-sided to get under 0.05, doubles the real chance of a false alarm.

“Fail to reject” is not “accept”. When p > α the test has found no convincing evidence against H₀, which can mean H₀ is true or that the sample was too small or too noisy to show a real difference. Report “no evidence of a difference at the 5% level”, with the interval, never “the test proved there is no difference”.

The z test when σ is known

If the population standard deviation $\sigma$ (sigma) is known, the standard error is $\sigma \div \sqrt{n}$ and the statistic follows the standard normal distribution: a z test. Kilnworth’s specification states $\sigma$ = 3 g. Taking that at its word, $z = (1{,}002.75 - 1{,}004) \div (3 \div \sqrt{40}) = -1.25 \div 0.4743 \approx -2.64$, and the one-sided p-value is =NORM.S.DIST(-2.635,TRUE) = 0.0042, again below 0.05.

The z test is less significant than the t test here because Kilnworth’s σ = 3 g is larger than Ana’s s = 2.66 g: its standard error is 0.474 g against 0.420 g, so the same 1.25 g shortfall is 2.64 standard errors rather than 2.98. A known σ is rare in business, and here it is only Kilnworth’s claim: if the machine has drifted, its spread may have changed too.

The one-sample t test uses Ana’s s and pays for the estimate with the t distribution’s heavier tails, so it needs no claim about σ. Some textbooks, OpenStax’s among them, switch from t to z once n reaches 30, using s in place of σ. Software uses t at every n, and so does this course: at n = 40 the normal curve would give 0.0015 for Ana’s t of −2.98 instead of 0.0025, the same decision with slightly overstated certainty.

P-values and critical values in Excel

Excel has no function for a one-sample t test: T.TEST compares two samples, as the comparing-means lesson shows. Calculate t from AVERAGE, STDEV.S and COUNT, then take the p-value from the function that matches the alternative.

T.DIST is Excel’s left-tailed t distribution (with TRUE for the cumulative probability; FALSE gives the height of the curve, which is not a probability) and T.DIST.RT its right tail. T.DIST.2T returns #NUM! for a negative x, so wrap the statistic in ABS. For Ana’s bags, =T.DIST.2T(ABS(-2.976),39) gives the two-sided p-value 0.0050, twice the one-sided 0.0025.

The critical value approach reaches the same decision without a p-value. The rejection region is the set of statistics extreme enough to reject H₀ at level α. For a one-sided lower-tail t test with 39 degrees of freedom it is t below =T.INV(0.05,39) = −1.685; for a two-sided test at 5% it is |t| above =T.INV.2T(0.05,39) = 2.023. Ana’s −2.98 lies beyond both. For z the familiar values are 1.645 (one-sided, 5%) and 1.96 (two-sided, 5%).

A p-value says more than the decision. A t of −2.98 and a t of −1.70 both lie beyond −1.685, so both reject at 5%, but the first would happen by chance about 1 time in 400 and the second about 1 time in 20: very different strengths of evidence. Report the p-value, not only “significant” or “not significant”.

Z.TEST’s one-tailed quirk

Excel’s Z.TEST(array, x, [sigma]) returns a one-tailed p-value, and always the upper tail. Microsoft defines it as 1 − NORM.S.DIST((AVERAGE(array) − x) ÷ (sigma ÷ √n), TRUE): the probability of a sample mean above the observed one if the population mean is x. For a sample mean below x the result is above 0.5. For Ana’s bags in B2:B41, =Z.TEST(B2:B41,1004,3) returns 0.9958.

The p-value for her lower-tail alternative is =1-Z.TEST(B2:B41,1004,3) = 0.0042, the z test’s result above. For a two-sided test Microsoft gives =2*MIN(Z.TEST(B2:B41,1004,3),1-Z.TEST(B2:B41,1004,3)), here 0.0084. Leave out sigma and Z.TEST uses the sample standard deviation but still the normal distribution, so it is still a z test, not a t test.

Testing a proportion

The same logic tests a proportion against a stated value $p_0$. Under H₀ the sample proportion $\hat{p}$ (“p-hat”) has standard error $\sqrt{p_0(1 - p_0) \div n}$, built from the null’s own value because the test asks how the data would behave if H₀ were true; a confidence interval uses $\hat{p}$, because it has no null value. The one-proportion z test statistic is $z = (\hat{p} - p_0) \div \sqrt{p_0(1 - p_0) \div n}$, compared with the standard normal. The normal model needs $np_0$ and $n(1 - p_0)$ both at least 10, a common rule of thumb (OpenStax’s version asks that “the sample proportion, p′, times the sample size is greater than 5” and that “1-p′ times the sample size is also greater than 5”), and independent trials.

In September the shop, web and app together and the basis of B01’s target, took 251 orders from 11,400 visits: a conversion rate of 2.20% against Ana’s 3.0% target. The applied example works the test: z ≈ −5.00, with a p-value of about 2.9 × 10⁻⁷. Two cautions apply to any test on visits. Visits are not fully independent trials, because one person can visit several times; and the test only says the shortfall is too large to be chance, not what caused it. Comparing the website’s August with its September needs a two-proportion test, which the proportions-and-chi-square lesson teaches.

Two ways to be wrong

A test decides between two hypotheses from incomplete information, so it can go wrong in two ways. A Type I error rejects a true null: a false alarm. A Type II error fails to reject a false null: a missed effect. The significance level α is the probability of a Type I error when H₀ is true; β (beta) is the probability of a Type II error for a particular true value of the parameter.

The two errors trade off. Lowering α to 0.01 makes false alarms rarer but makes a real underfill harder to detect, raising β. Which error costs more is a business judgement: a false complaint costs some goodwill with a supplier and an engineer’s visit, while a missed underfill costs Ana’s customers coffee and, as more bags fall under the label, her reputation. Those costs, not convention, should set α.

Power, and what raises it

The power of a test is $1 - \beta$: the probability that it rejects H₀ when a particular alternative is true. It depends on the size of the true difference $\delta$ (delta), the spread $\sigma$, the sample size $n$, the significance level and whether the test is one- or two-sided. For a one-sided z test, power $= \Phi(|\delta| \div (\sigma \div \sqrt{n}) - z_{1-\alpha})$, where $\Phi$ is the standard normal cumulative probability, NORM.S.DIST(…, TRUE).

Suppose the machine really averages 1,003 g, 1 g below the setting. With $\sigma$ = 3 g, n = 40 and α = 0.05 one-sided, the standard error is 0.474 g, so the true mean sits $1 \div 0.474 \approx 2.11$ standard errors below the null value, and power $= \Phi(2.11 - 1.645) = \Phi(0.46) \approx 0.68$. Forty bags would catch a 1 g underfill about two times in three. They would catch a 0.5 g underfill only 28% of the time, and a 2 g underfill 99% of the time.

• A larger sample: the standard error shrinks with √n.
• A larger true difference: big effects are easier to see than small ones.
• Less noise: a smaller σ, from more precise scales or more uniform conditions.
• A larger α: a lenient threshold rejects more often, at the price of more false alarms.
• A one-sided test, when the effect lies in the direction tested; it can never detect one in the other direction.

Turned round, the formula gives the sample size for a chosen power: $n = ((z_{1-\alpha} + z_{1-\beta})\,\sigma \div \delta)^2$, rounded up, with $z_{1-\alpha/2}$ in place of $z_{1-\alpha}$ for a two-sided test. Use it before collecting data, from the smallest difference worth detecting; the challenging example plans Ana’s next check. The formula treats $\sigma$ as known, and for a t test the NIST/SEMATECH handbook repeats the calculation once with t values, which adds a bag or two.

Statistical and practical significance

A result is statistically significant when its p-value is at or below α. That says the effect is unlikely to be chance alone; it says nothing about whether the effect is large enough to matter. The American Statistical Association’s 2016 statement on p-values makes this its fifth principle: a p-value measures neither the size of an effect nor its importance.

With enough data, trivial differences become significant. If Ana weighed 4,000 bags and found a mean of 1,003.9 g, then with $\sigma$ = 3 g, $z = -0.1 \div (3 \div \sqrt{4{,}000}) \approx -2.11$ and p ≈ 0.018: significant at 5%, for a tenth of a gram. In the other direction, a small sample can miss an important effect. Report the estimate and its interval, and judge practical significance by what the difference does to the business.

For the bags, practical significance comes through the label. A machine averaging 1,002.75 g with σ = 3 g would put about 18% of bags under 1,000 g, =NORM.DIST(1000,1002.75,3,TRUE), roughly twice the 9.1% of the specification. A shortfall of 1.25 g is about 0.1% of the coffee in a bag, but it doubles the share of light bags, and light bags are what customers notice.

Intervals and tests

A two-sided test at level α and a (1 − α) confidence interval are two views of one calculation: a two-sided t test at 5% rejects H₀: μ = μ₀ exactly when μ₀ lies outside the 95% t interval. Ana’s 95% interval for the mean fill is 1,001.90 to 1,003.60 g. Both 1,004 g and 1,000 g lie outside it, so two-sided tests reject both (p = 0.0050 and p < 0.0001), while 1,003 g lies inside and would not be rejected.

The interval says more than the test. The machine’s mean is plausibly between 0.40 g and 2.10 g below its setting, which is the range Ana should keep in mind when she judges whether the shortfall matters. A one-sided test at 5% matches a one-sided 95% bound instead: here the mean is below 1,003.46 g with 95% confidence.

P-hacking and multiple testing

Every test at α = 0.05 has a 5% chance of a false alarm when H₀ is true. Run many tests and false alarms become likely: with 20 independent tests of true nulls, the chance of at least one p ≤ 0.05 is 1 − 0.95²⁰ ≈ 0.64, and one false positive is expected. This is the problem of multiple testing. Testing every metric on a dashboard each month, or every customer segment, will produce “significant” findings that are noise.

P-hacking is the deliberate or careless version: trying analyses until one gives p below 0.05, and reporting that one. Its common forms are testing many outcomes or subgroups, adding data and retesting until the result crosses the line, dropping awkward values, and choosing one-sided after seeing the data. Each makes the reported p-value smaller than the real chance of a false alarm.

In 2016 the American Statistical Association published a statement on p-values (Wasserstein and Lazar) with six principles. In paraphrase, except the fifth, which is quoted:

1. A p-value can show how incompatible the data are with a stated statistical model, such as one in which H₀ is true.
2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data arose by chance alone.
3. Scientific conclusions and business or policy decisions should not rest only on whether a p-value passes a threshold such as 0.05.
4. Proper inference requires full reporting and transparency: how many hypotheses were explored, which analyses were run and every p-value calculated, not only the significant ones.
5. “A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.”
6. By itself a p-value is not a good measure of evidence about a model or hypothesis: a p-value near 0.05 is only weak evidence against the null, and a large p-value is not evidence for it.

The defences are habits. Write down the question, the hypotheses, α and the analysis before looking at the results; report every test you ran; treat a surprising result from one of many tests as a lead to check with new data. When you must test several things at once, make each test stricter: the simplest adjustment, Bonferroni’s, tests each of k hypotheses at α ÷ k, and the anova lesson explains the family-wise error rate behind it.