Proportions and chi-square tests

In depth

The two-proportion z test

Two independent samples of sizes $n_1$ and $n_2$ contain $x_1$ and $x_2$ successes, giving sample proportions $\hat{p}_1 = x_1 \div n_1$ and $\hat{p}_2 = x_2 \div n_2$. To test $H_0\colon p_1 = p_2$, the standard error uses the pooled proportion $\bar{p} = (x_1 + x_2) \div (n_1 + n_2)$, because under the null hypothesis both samples estimate the same p and the combined sample estimates it best. The test statistic divides the gap between the sample proportions by that standard error:

z is compared with the standard normal distribution: two-sided for $H_1\colon p_1 \neq p_2$, one-sided for $p_1 \gt p_2$ or $p_1 \lt p_2$, a choice made before looking at the data.

Its conditions are those of the one-proportion test, applied to each sample:

• Independence. The two samples are independent of each other, and the observations within each are independent: random samples, randomly assigned groups or, for a process such as Lantern’s visits, trials that don’t influence one another.
• Large enough counts. At least 10 successes and 10 failures in each sample, so that the normal approximation holds. OpenStax’s Introductory Business Statistics asks for at least 5; 10 is the more cautious rule and the one this course uses for intervals. Lantern’s counts (224 and 7,476 in August, 158 and 8,642 in September) pass easily.
• A small fraction of the population. When samples are drawn without replacement from a finite population, OpenStax adds that the population should be at least 10 to 20 times the size of the sample.

Lantern’s visits bend the independence condition: one person can visit several times in a month, and their visits are related. Related trials can make the formula’s standard error too small, so a borderline z would deserve caution. A z of 4.75 is not borderline.

Excel has no function for a two-proportion test, so build it from cells. With the orders in B2:C2 and the visits in B3:C3, the pooled proportion is =(B2+C2)/(B3+C3); with that in B6, the standard error is =SQRT(B6*(1-B6)*(1/B3+1/C3)); and with z in B8, the two-sided p-value is =2*(1-NORM.S.DIST(ABS(B8),TRUE)). For a one-sided alternative, =1-NORM.S.DIST(B8,TRUE) gives P(Z > z), the p-value for $p_1 gt p_2$, and =NORM.S.DIST(B8,TRUE) gives P(Z < z), the p-value for $p_1 lt p_2$.

A confidence interval for the difference

An interval estimates how big the difference is, not only whether it is zero. It doesn’t assume the two proportions are equal, so it uses each sample’s own proportion in an unpooled standard error:

Here $z^* \approx 1.96$ for 95%. For Lantern the unpooled standard error is 0.002382, the margin $1.96 \times 0.002382 \approx 0.00467$, and the interval $0.01114 \pm 0.00467$: 0.65 to 1.58 percentage points.

Because the test and the interval use slightly different standard errors, they can disagree when a result is borderline: an interval that just excludes 0 alongside a p-value just above 0.05. Far from the line, as here, they agree. Report the interval with the test: “conversion fell by 1.1 points (95% CI 0.6 to 1.6)” says both that the fall is unlikely to be chance and how large it might be. In relative terms September converted at 62% of August’s rate, a fall of 38%.

The chi-square distribution

A chi-square distribution with k degrees of freedom is the distribution of the sum of the squares of k independent standard normal variables. It is never negative and it is skewed to the right; its mean equals its degrees of freedom and its standard deviation is √(2k). As the degrees of freedom grow, it moves right, spreads out and becomes more symmetric.

Why does the χ² statistic follow it? When counts are large, each standardised gap $(O - E) \div \sqrt{E}$ behaves roughly like a standard normal value, so their squares add up like a chi-square variable. The gaps are not all free, though: they must add to 0, because the observed and expected counts share a total, and every quantity estimated from the data adds another constraint. Each constraint removes one degree of freedom. The approximation is poor when expected counts are small, which is where the rule of at least 5 comes from.

Excel’s functions: =CHISQ.DIST.RT(x,deg_freedom) returns the right-tail probability P(χ² > x), the p-value of a chi-square test; Microsoft’s example, =CHISQ.DIST.RT(18.307,10), returns 0.0500006. =CHISQ.INV.RT(probability,deg_freedom) goes the other way and returns the critical value: =CHISQ.INV.RT(0.05,6) ≈ 12.59 for the weekday test. =CHISQ.DIST(x,deg_freedom,TRUE) returns the left-tail probability, the wrong one for a p-value. Both right-tail functions truncate a deg_freedom that isn’t a whole number.

Goodness-of-fit tests

A goodness-of-fit test asks whether one categorical variable follows a stated distribution: a planned product mix, equal rates per day, a Poisson model. The steps:

1. State the model as a probability for each of the k categories. The probabilities come from the null hypothesis, not from the counts being tested.
2. Multiply by the sample size to get the expected counts, $E_i = n p_i$. They needn’t be whole numbers.
3. Check that every expected count is at least 5. If one isn’t, merge it with a neighbouring category, and make sure the categories between them cover every possible value.
4. Calculate χ² and its degrees of freedom, $k - 1 - m$, where m is the number of the model’s parameters estimated from the same data.
5. Find the right-tail p-value with CHISQ.DIST.RT, and state the conclusion in context.

The rule of 5 is a convention. OpenStax states it, and Microsoft’s CHISQ.TEST page says that some statisticians suggest it; NIST’s e-Handbook asks only for a sufficient sample size and warns that the result depends on how the data are binned. OpenStax calls the goodness-of-fit test “almost always right-tailed”: large values of χ² are the evidence against the model. The rare exception is a χ² far below its degrees of freedom, which can mean data that fit too well, for example because they were tidied.

Estimated parameters cost degrees of freedom. To test whether orders per day follow a Poisson distribution, Ana must estimate λ from the same 122 days, and a model fitted to the data matches them better than a model fixed in advance. To allow for that, the statistic is compared with a chi-square distribution with one degree of freedom fewer. NIST’s handbook gives the general rule: k − c degrees of freedom, where c is the number of estimated parameters plus 1.

In Excel, =CHISQ.TEST(actual_range,expected_range) returns the p-value directly from two ranges of the same size. For a single column of k categories it uses k − 1 degrees of freedom, which is right only when no parameter was estimated. Otherwise calculate χ² yourself, for example =SUMPRODUCT((B2:B7-D2:D7)^2/D2:D7) with the observed counts in B and the expected counts in D, and take =CHISQ.DIST.RT with the correct degrees of freedom.

The test of independence

For a table with r rows and c columns, the null hypothesis is that the row variable and the column variable are independent, and the alternative is that they are associated. Independence means P(row i and column j) = P(row i) × P(column j), as in Probability. Estimating each probability by its share of the grand total n and multiplying by n gives the expected count $E_{ij} = R_i C_j \div n$: the row total times the column total over the grand total.

The degrees of freedom are $(r - 1)(c - 1)$: with the row and column totals fixed, once $(r - 1)(c - 1)$ cells are filled in, the rest are determined. Lantern’s five regions by subscription status, a 5 × 2 table, have 4. The conditions:

• The table counts individuals (orders, customers, bookings), each in exactly one cell. Percentages and averages can’t go into a chi-square test.
• The observations are independent. Several orders from one customer bend this condition, as repeat visits did for conversion.
• Every expected count is at least 5. OpenStax: “The expected value inside each cell needs to be at least five in order for you to use this test.” If one isn’t, merge categories that belong together or collect more data.

In Excel, build the expected table first. With the observed counts in B2:C3, the row totals in D2:D3, the column totals in B4:C4 and the grand total in D4, type =$D2*B$4/$D$4 in the top-left expected cell and fill it across and down. =CHISQ.TEST(B2:C3,B7:C8) then returns the p-value with (r − 1)(c − 1) degrees of freedom, or #N/A if the two ranges hold different numbers of values. It doesn’t return χ² itself; for a report, calculate it with SUMPRODUCT.

When each row is a separate sample, for example a survey of 200 web customers and 200 app customers, the same calculation is called a test of homogeneity: are the column proportions the same in every row? The arithmetic and the conditions don’t change.

Two-by-two tables: χ² = z²

A 2 × 2 table holds two proportions, so the chi-square test and the two-proportion z test can be run on the same counts. They agree exactly. For Lantern’s channels the z test of the app’s subscription share against the web’s gives $z = (0.5758 - 0.3561) \div 0.0661 \approx 3.32$, and $3.32^2 \approx 11.03 = \chi^2$, with the same p-value, 0.0009. The chi-square test is the two-sided z test; use the z test when you want a one-sided alternative or an interval, here a gap of 22.0 points with a 95% interval of 9.3 to 34.7 points.

Some textbooks and some software apply a continuity correction to 2 × 2 tables, which makes χ² a little smaller. Excel’s CHISQ.TEST works from the formula above without one, and so does this lesson, which is why its results match the z test.

How strong, and what it doesn’t show

A p-value measures evidence, not size. With thousands of observations a trivial association is statistically significant; with a few dozen a large one may not be. Report an effect size alongside: the percentages being compared and, for a larger table, Cramér’s V, $V = \sqrt{\chi^2 \div (n \times \min(r - 1, c - 1))}$, which runs from 0 (counts that match independence exactly) to 1 (complete association). Lantern’s channel and subscription give V ≈ 0.22, its ratings by channel V ≈ 0.002, and its regions by subscription V ≈ 0.34, driven by the East’s 75% subscription share against 32% to 44% elsewhere.

A non-significant result is not evidence of no difference. The rated orders’ five-star shares differ by only 0.2 points, but the 95% interval for the difference runs from −15.5 to +16.0 points: 154 ratings can’t rule out a sizeable gap either way.

And association is not causation. App orders were subscriptions more often, but customers chose their channel: committed regulars may be the ones who install an app and the ones who subscribe, a confounding variable that a chi-square test can’t see. The test says the pattern is unlikely to be chance; whether the app causes subscriptions needs a comparison in which customers don’t choose, such as a randomised trial. Finally, every test carries its α risk: run chi-square tests on twenty cross-tabulations of the same orders and about one will be significant at the 5% level by chance alone, the multiple testing problem from Hypothesis testing.