Sampling methods

In depth

Probability designs

B01 and B04 introduced the four basic random designs; the question for a researcher is which suits the frame, the budget and the estimates needed. A simple random sample needs only a complete list. A systematic sample takes every kth unit from a random start, which is easy from a list and spreads the sample across it, but can go wrong if the list has a cycle of the same length. Lantern’s list is in sign-up order; every third person from a random start among the first three would give 2,000 invitations.

A stratified sample needs the stratum of every unit on the frame before sampling, which Lantern’s list provides. It guarantees each group its share of the sample and is more precise than a simple random sample of the same size when the strata differ on what is measured. A cluster sample selects whole groups, such as stores, streets or sign-up batches, and takes everyone in them. It is cheaper when travelling to or listing every unit is expensive, and less precise, because members of a cluster tend to resemble each other.

A multistage sample combines these: it samples clusters, then units within them, sometimes in more than two stages. A national shopper survey might choose towns at random, then supermarkets in each chosen town, then, at randomly chosen times, every kth shopper leaving each chosen supermarket. It needs a list only of the units at each stage (towns, then those towns’ supermarkets), not of every shopper in the country, which is why face-to-face surveys of large populations are usually multistage.

The design effect

The design effect compares a design with simple random sampling: it is the variance of an estimate under the design divided by its variance under a simple random sample of the same size. A design effect of 2 means the design needs twice the sample to match a simple random sample’s precision; its effective sample size is $n \div \text{deff}$. Proportionate stratification usually gives a design effect a little below 1; disproportionate allocation can push it above 1 (Orrinsby, below); clustering gives one above it.

For clusters of equal size b the design effect is about $1 + (b - 1)\rho$, where ρ, the intraclass correlation, measures how alike members of the same cluster are (0 if no more alike than strangers, 1 if identical). Because ρ is multiplied by the cluster size, even a small one hurts: Marrowby Buses (fictional) surveying all 25 passengers on each of 40 buses, with ρ = 0.04, has a design effect of $1 + 24 \times 0.04 = 1.96$, so its 1,000 passengers carry the precision of about 510 independent ones.

Allocating a stratified sample

Proportional allocation gives each stratum the same share of the sample as it has of the frame: $n_h = n \times N_h \div N$. Every unit then has the same chance of selection, and the sample is self-weighting: its plain percentages estimate the population’s. For Lantern, $2{,}004 \times 520 \div 6{,}000 = 173.68$, and likewise 334.00, 50.10, 1,446.22 for the other three strata. The parts must be whole and add to 2,004: rounding down gives 2,003, and the largest-remainder method gives the missing invitation to the stratum with the largest fraction left over (subscribers, .68), which is how 174 / 334 / 50 / 1,446 was reached.

In Excel, put the stratum sizes in B2:B5 and the sample size in B8, and fill =$B$8*B2/SUM($B$2:$B$5) down for the unrounded allocation. =ROUND($B$8*B2/SUM($B$2:$B$5),0) usually works but can leave the total one out: Brindlecombe Garden Centre’s example below rounds to 401. Check the total, and adjust the stratum with the largest remainder by hand.

Disproportionate allocation deliberately samples some strata at higher rates than others, usually to read a small group separately. The units then have unequal chances of selection, so the strata must be combined with population weights, not by pooling the sample: the next section shows how. Neyman allocation is the allocation that minimises the standard error of the overall estimate for a fixed n: $n_h \propto N_h S_h$, more sample where strata are larger or more variable. For a proportion, $S_h = \sqrt{p_h(1 - p_h)}$ barely changes between 0.3 and 0.7, so Neyman allocation is close to proportional unless some strata have shares near 0 or 1.

The stratified estimate and its standard error

Whatever the allocation, the stratified estimate of a proportion weights each stratum’s sample proportion by the stratum’s share of the population: $\hat{p}_{st} = \sum W_h \hat{p}_h$, with $W_h = N_h \div N$. Its standard error adds the strata’s variances, each weighted by $W_h^2$ and corrected for the share of the stratum sampled. Under proportional allocation $\hat{p}_{st}$ equals the sample’s overall proportion; under disproportionate allocation the two differ, and only the stratified estimate is right.

Orrinsby Software (fictional), worked in full below, sampled its three plans at very different rates. Pooling its 500 replies gives 29.4% who would pay for a new feature; the stratified estimate is 16.4%, with a standard error of 1.83 points. The pooled figure counts each Enterprise reply as if it stood for as many customers as a Free reply, when it stands for a tenth as many (4 customers against 40). In Excel, with the stratum sizes in B2:B4, sample sizes in C2:C4 and “yes” counts in D2:D4, the estimate is =SUMPRODUCT(B2:B4,D2:D4/C2:C4)/SUM(B2:B4).

Non-probability designs

Five kinds are common in market research, and none gives each unit a known chance of selection.

• Convenience sample (B01): whoever is easiest to reach, such as shoppers passing a stall.
• Judgement sample: units the researcher picks as typical or informative, such as the five stores a manager thinks represent the chain. OpenStax names it alongside convenience sampling.
• Quota sample: interviewers fill set numbers in groups defined by easily observed traits, such as 50 women and 50 men in each age band, choosing whom to approach themselves. The AAPOR task force describes it as a form of sample matching and recalls that the quota-based polls of the 1948 U.S. presidential election went wrong. The quotas match the population on the quota traits only; within each quota, the interviewer’s choices decide who is asked.
• Snowball sample: participants recruit further participants from among their contacts, useful for groups with no list, such as owners of a rare car. Its make-up depends on who started the chain and whom they know.
• Online panels and river samples: an opt-in panel is a pool of volunteers who signed up to take surveys, usually for rewards; a river sample, in AAPOR’s words, is a web-based opt-in approach that recruits respondents through invitations on websites (pop-ups, banners and links). Both are self-selected: the AAPOR task force notes that participation rates in surveys of opt-in panels are often in single digits.

Quota sampling is the one most often mistaken for stratification, because both fix numbers per group. The difference is in the last step: a stratified sample draws at random within each stratum from a list, whereas a quota sample lets interviewers fill each cell with whoever they find. People who are at home, approachable or online at the right moment fill the quotas, and nothing guarantees they resemble the rest of their group.

Non-probability samples have honest uses: pretests, qualitative research, exploring hard-to-reach groups, tracking changes over time with a method held constant, and fast reads where a rough answer is enough. What they can’t do is support a margin of error on their own. Report them as what they are (“of 400 panel members who took part”), describe how they were recruited and weighted, and treat their levels with caution.

How big a sample?

B04 gave the size for a proportion: $n = z^2 p(1 - p) \div E^2$, with p = 0.5 when nothing is known, so ±4 points at 95% needs 601 completed questionnaires. Three adjustments turn that into a plan. When the sample is a large share of a small population, the finite population correction reduces the need to $n = n_0 \div (1 + (n_0 - 1) \div N)$. The number of invitations is the completes needed divided by the expected response rate: $601 \div 0.3 = 2{,}003.3$, so 2,004. And if a subgroup must be read separately, the size must be worked out for that subgroup, not for the whole.

The last adjustment is the one most often missed. A proportional design gives each stratum its share of the completes, so small strata get few. At the planned 30% response, Lantern’s 50 workshop invitations would yield about 15 completes, a margin of about ±24 points for that group, too wide to say anything about workshop participants on their own. Reading every stratum to ±10 points needs about 97 completes each before the correction (fewer in the small strata), which means a disproportionate design and weighting afterwards.

Two warnings go with every sample size. It is a number of usable completes, so the invitations depend on a response rate that is itself a guess. And it controls sampling error only: a large sample from the wrong frame, or one in which the people who reply differ from those who don’t, is precisely wrong. Who replies is the subject of the next lesson.