Survey error · M04 · Sunflower Intelligence

Before you start

You will be able to

Distinguish the components of total survey error (coverage, sampling, non-response, adjustment, measurement and processing) and identify the step at which each enters a survey.
Classify survey cases by AAPOR’s final dispositions and calculate the response rates RR1 and RR2.
Calculate non-response bias from the response rate and the difference between respondents and non-respondents, and explain why a response rate alone doesn’t measure bias.
Calculate post-stratification weights, a weighted estimate, the Kish effective sample size and the design effect of unequal weights, and explain what weighting can’t fix.
Evaluate stated purchase intentions against behaviour, and identify measurement and processing errors from wording, mode, memory and data handling.

Review first

Collecting data: sources, samples and bias — The four biases (sampling, non-response, survivorship, measurement), the response rate and the bounds non-response allows are built on here.
Sampling methods — Frames, coverage, the stratified estimate and the design effect are used throughout.
Questionnaire design — Wording, response styles, order and mode are the sources of measurement error.

Fictional case A fictional coffee business founded by Ana Lim. It starts as a catering and barista-training service, opens its own cafés and an online shop for beans, and plans a canned cold brew.

The idea

Fictional case

Ana’s survey closed on 16 February 2027. Of the 2,004 people invited, 516 completed the questionnaire: 25.7%, not the 30% she had planned for, so the margin of error for a share near 50% is ±4.3 points rather than ±4. Response was far from even: about half the subscribers replied, against a fifth of the newsletter-only group. The 516 completes give a Net Promoter Score of +27.5. Ana’s question is the one this lesson answers: how far can she trust it?

Total survey error

Total survey error is the idea that a survey estimate can be wrong for many reasons, and that they should be designed against and judged together rather than one at a time. Groves and Lyberg (2010), reviewing its history, trace it back to Deming, who in 1944 listed thirteen factors affecting survey results, from non-response, sampling and interviewer effects to processing errors and timeliness, and note that the term is not tightly defined: different authors include different components. A widely used version, from Groves and colleagues’ textbook Survey Methodology (2004), follows the two paths an estimate travels.

Total survey error: an estimate is only as good as both paths. On the representation side the question is who answered; on the measurement side, what their answers mean. The margin of error covers sampling error alone.

On the representation side, coverage error arises from the gap between the target population and the frame (undercoverage and overcoverage, from Sampling methods); sampling error from studying a sample rather than everyone; non-response error from the difference between those sampled and those who answer; and adjustment error from weighting that corrects the wrong things. On the measurement side, a question may not capture the construct at all (validity), answers may depart from the truth (measurement error), and recording, coding and editing may introduce mistakes (processing error).

B01’s four biases fit inside this map. Sampling bias and survivorship bias are coverage and selection problems on the representation side; non-response bias is the systematic part of non-response error; measurement bias is the systematic part of measurement error. What the map adds is that the errors compound, and that fixing one (a bigger sample) does nothing for another (a biased frame).

Each error has a systematic part, bias, and a random part, variance. Groves and Lyberg recall that Hansen and colleagues wrote the total as the mean squared error: $\text{MSE} = \text{bias}^2 + \text{variance}$ , plus terms for how errors interact. They also note that most surveys routinely measure only one component, the sampling variance. That is the margin of error, and it is why a margin of ±4.3 points says nothing about whether the estimate is 10 points too high.

Response rates by the book

The American Association for Public Opinion Research (AAPOR) publishes Standard Definitions, now in its tenth edition (2023), so that response rates mean the same thing from one survey to the next. Every case drawn into the sample is given a final disposition in one of four groups: interviews (complete, or partial); eligible cases that weren’t interviewed (refusals and break-offs, non-contacts and others); cases of unknown eligibility; and cases that turn out not to be eligible, which are left out of the rates altogether.

Researchers must define in advance what counts as complete and partial. AAPOR gives examples rather than one rule: in one, more than 80% of the questions answered is a complete and 50% to 80% a partial. Ana set her rule before fieldwork (fictional): all five closed questions (Q1 to Q5) answered is a complete, whether or not the open Q6 was filled in; three or four, a partial; fewer than three, a break-off, counted with refusals.

Lantern’s February 2027 survey: final dispositions by stratum
Stratum	Invited	Complete (I)	Partial (P)	Refused or broke off (R)	Bounced (NC)	No reply (eligible)
Subscribers	174	89	11	1	6	67
Other shop customers	334	103	10	4	9	208
Workshop participants	50	27	0	1	0	22
Newsletter only	1,446	297	35	21	42	1,051
All	2,004	516	56	27	57	1,348

RR1, AAPOR’s strictest response rate, counts only completes: $\text{RR1} = I \div (I + P + R + NC + O + U)$ , 516 ÷ 2,004 = 25.7%. RR2 counts partials as responses too: $(516 + 56) \div 2{,}004$ = 28.5%. Everyone on Lantern’s list belongs to the population it was drawn for, so no case is ineligible, and every bounced or unanswered invitation stays in the denominator.

AAPOR’s RR3 and RR4 allow an estimated share of the unknown-eligibility cases to be removed from the denominator, which matters for samples such as random phone numbers, where many may not belong to anyone eligible. Always say which rate you report. By stratum, RR1 was 51.1%, 30.8%, 54.0% and 20.5% for subscribers, shop customers, workshop participants and newsletter-only readers, and RR2 57.5%, 33.8%, 54.0% and 23.0%.

Fewer completes than planned means a wider margin: with 516 instead of 601, ±4.31 points instead of ±4.00. That loss is visible and modest. The serious question is about the 1,488 people who didn’t complete.

Why a response rate is not a measure of bias

B01 combined respondents and non-respondents as a weighted average. Rearranged, the same identity gives the bias of the respondents’ figure: $\bar{y}_r - \bar{y} = (1 - \text{RR})(\bar{y}_r - \bar{y}_{nr})$ , the share who didn’t respond times the difference between respondents and non-respondents. Bias needs both: a low response rate with no difference does no harm, and a high one with a large difference still misleads.

This is why AAPOR states that response rate information alone is insufficient for determining how much non-response error exists. The U.S. National Research Council’s 2013 review of non-response cites Groves and Peytcheva’s compilation of 59 studies, which found very little correlation between non-response rates and their measures of bias, and concludes that a falling response rate means only that the potential for bias has increased. Bias belongs to each estimate, not to the survey: the same non-respondents can distort one question badly and another hardly at all.

Ana can’t see her non-respondents’ answers, but she can see who they are. Subscribers were 8.7% of the list and 17.2% of the completes; newsletter-only readers were 72.2% of the list and 57.6% of the completes. The segments also answered very differently: Net Promoter Scores of +62, +53, +41, +7 for subscribers, shop customers, workshop participants and newsletter-only readers. A respondent pool tilted towards subscribers will flatter Lantern, and that tilt is the one thing she can correct.

In depth

Weighting to the population

Post-stratification gives every respondent in group h the survey weight $w_h = N_h \div n_h$ : the number of people on the frame that each respondent stands for. Lantern’s weights are 520 ÷ 89 = 5.84; 1,000 ÷ 103 = 9.71; 150 ÷ 27 = 5.56; 4,330 ÷ 297 = 14.58. Each newsletter-only respondent now counts about two and a half times as much as each subscriber, and the weighted respondents add back up to the 6,000 on the list.

A weighted share is $\sum w_i y_i \div \sum w_i$ , which, with one weight per stratum, equals the stratified estimate from Sampling methods: $\sum W_h \hat{p}_h$ . Weighted, 40.5% of Lantern’s contacts are promoters and 20.1% detractors, so the weighted Net Promoter Score is +20.4, against +27.5 unweighted. Seven points of the unweighted score came from the over-representation of the three customer segments.

Lantern’s post-stratification weights (respondents are February’s completes)
Stratum	On the list (N_h)	Respondents (n_h)	Weight N_h ÷ n_h	Share of list	Share of respondents
Subscribers	520	89	5.84	8.7%	17.2%
Other shop customers	1,000	103	9.71	16.7%	20.0%
Workshop participants	150	27	5.56	2.5%	5.2%
Newsletter only	4,330	297	14.58	72.2%	57.6%
All	6,000	516		100.0%	100.0%

What weighting costs

Unequal weights make an estimate less precise, because a few heavily weighted respondents move it more. Kish’s rule of thumb measures the loss: the effective sample size is $(\sum w_i)^2 \div \sum w_i^2$ , and the design effect of the weights is $n \div n_{\text{eff}} = 1 + \text{CV}^2(w)$ , one plus the squared coefficient of variation of the weights. For Lantern’s 516 completes, $n_{\text{eff}} = 469.3$ and the design effect 1.099: the weighting costs about 47 completes’ worth of precision.

Zipf and Valliant’s guide to design effects warns that Kish’s formula rests on restrictive assumptions (equal variances in every stratum) and can mislead when strata differ, or when response rates differ across groups. Both hold here. Computed directly with the stratified standard error from Sampling methods, the weighted share of promoters has a standard error of 2.15 points before the finite population correction (2.07 with it), against 2.27 from Kish’s effective sample size: because the segments differ so much in their share of promoters, stratifying wins back most of what the unequal weights cost. Use n_eff as a quick warning; use the design’s own standard error when it matters.

What weighting can’t fix

Post-stratification makes the respondents match the population on the weighting variable, and nothing else. It removes bias only to the extent that, within each weighting group, respondents answer like the non-respondents in that group: in B02’s terms, that the missing answers are MAR given the stratum. If, within every stratum, the people who liked Lantern were keener to reply, weighting by stratum reproduces their enthusiasm at the right scale.

The same respondents distort different questions by different amounts. For Q2, weighting moves the share who would definitely or probably buy Cold Brew from 42.2% to 38.2%, within about a point of the list’s true 39.3%: willingness to try Cold Brew had little to do with who replied, once the strata were balanced. Recommending Lantern had a great deal to do with it. Run the two samplers below with non-response and weighting switched on and compare.

Weighting has the same limits on a larger scale. Pew Research Center weighted large online opt-in samples on demographic and political variables by several methods: average estimated bias across 24 benchmarks fell from 8.4 points unweighted to no lower than about 6 points, and in Pew’s words, “even the most effective adjustment procedures were unable to remove most of the bias.” Which variables went into the weights mattered more than the method used.

What helps is evidence about the non-respondents and better variables to weight on. Follow up a random sample of non-respondents by another mode, as B01’s hotel did, and use their answers to estimate the gap directly. Compare respondents with non-respondents on everything the frame already records: for Lantern, order history, subscription length and sign-up channel. Weight on the variables that predict both replying and the answer. And raise response where it is lowest, with reminders, a shorter questionnaire or a small incentive, while remembering that a higher rate helps only if it brings in people unlike those already responding.

Measurement error: wording, mode and memory

On the other side of the map, answers can be wrong however representative the respondents. Questionnaire design covered the main sources: wording that leads, loads or confuses; response styles such as acquiescence and social desirability; order effects; and the mode, which changes how social pressures and option orders work. Memory adds two more. People forget, and they misplace events in time, pulling them into or out of the period asked about, which is why Q3 asks about last week, not last month.

Item non-response, a respondent skipping some questions, sits between the two sides: the person answered the survey but not the question. It is distinct from unit non-response, not answering at all, and is handled in the analysis (module 4) by reporting bases and coding missing answers rather than treating blanks as zeros.

Stated intentions are not purchases

Purchase-intent questions are measurement with a special problem: they ask people to predict their own future behaviour. Morwitz, Steckel and Gupta’s (2007) analysis of studies relating intentions to later purchases found intentions more strongly related to purchases for existing products than for new ones, for durable goods than for non-durables, over short horizons than long ones, and for specific brands or models than for whole categories. Sun and Morwitz (2010) describe respondents who exaggerate their demand, producing a positive intention bias that is likely in new-product surveys; in one of their data sets, 15.3% of respondents said they would buy a personal computer within a year, but only 5.7% did.

Lantern’s Q2 is about a new, non-durable product, asked with a three-month horizon and a named brand: two conditions that weaken the link and two that help. The weighted 38.2% who say they would definitely or probably buy Cold Brew is a measure of interest among Lantern’s contacts, not a forecast of buyers. Module 3’s market sizing turns intent into an estimate of purchases only with an explicit conversion assumption, and tests how much the answer depends on it.

Processing error

Errors also enter after the answers arrive. A partial is counted as a complete; a stratum code is attached to the wrong respondent when the list is merged with the survey export; a scale is keyed in reverse; weights are applied to the wrong rows; an open answer is coded by one person differently from another. Most of these are systematic: they push estimates one way rather than averaging out. The defences are routine and dull: a written data plan, checks that weighted totals reproduce the frame’s, a second coder for open answers with inter-coder agreement checked, and keeping the raw export untouched alongside every edited version.

Formulas

AAPOR response rates RR1 and RR2

\text{RR1} = \dfrac{I}{(I + P) + (R + NC + O) + U} \qquad \text{RR2} = \dfrac{I + P}{(I + P) + (R + NC + O) + U}

$I$: Complete interviewscases
$P$: Partial interviewscases
$R$: Refusals and break-offscases
$NC$: Non-contacts (for an e-mail survey, invitations that bounced)cases
$O$: Other eligible non-interviewscases
$U$: Cases of unknown eligibility. On a list where everyone is known to be eligible there are none: AAPOR (10th edition, section 1) counts unanswered invitations as eligible non-respondents, in the denominator either waycases

Assumes. Cases found to be ineligible are excluded from the denominator. The definitions of complete and partial are set before fieldwork.

RR3 and RR4 replace U with e × U, where e is the estimated share of unknown cases that are eligible (AAPOR, Standard Definitions, 10th edition, 2023).

Non-response bias of a respondent mean or share

\bar{y}_r - \bar{y} = (1 - \text{RR})\,(\bar{y}_r - \bar{y}_{nr})

$\bar{y}_r$: The mean or share among respondents
$\bar{y}_{nr}$: The mean or share among non-respondents
$\bar{y}$: The mean or share among everyone sampled
$\text{RR}$: The response rate

Assumes. Treats each sampled person as a fixed respondent or non-respondent; it is B01’s weighted average of the two groups, rearranged.

ȳ_nr is unknown unless non-respondents are followed up; the formula shows why a response rate alone can’t give the bias.

Post-stratification weights and a weighted estimate

w_h = \dfrac{N_h}{n_h}, \qquad \bar{y}_w = \dfrac{\sum_i w_i y_i}{\sum_i w_i} = \sum_h \dfrac{N_h}{N}\,\bar{y}_h

$w_h$: The weight of each respondent in group hpeople per respondent
$N_h$: The number in group h in the population (on the frame)people
$n_h$: The number of respondents in group hpeople
$w_i,\ y_i$: Respondent i’s weight and answer (1 or 0 for a share)
$\bar{y}_h$: The mean or share among group h’s respondents
$N$: The population sizepeople
$\bar{y}_w$: The weighted estimate

Assumes. Population counts N_h are known. The estimate is unbiased only if respondents answer like non-respondents within each group.

Kish effective sample size and the design effect of weighting

n_{\text{eff}} = \dfrac{\left(\sum_i w_i\right)^2}{\sum_i w_i^2}, \qquad \text{deff}_w = \dfrac{n}{n_{\text{eff}}} = 1 + \text{CV}^2(w)

$w_i$: Respondent i’s weight
$n$: The number of respondentspeople
$n_{\text{eff}}$: The effective sample sizepeople
$\text{CV}(w)$: The coefficient of variation of the weights (their standard deviation, dividing by n, ÷ their mean)
$\text{deff}_w$: The design effect of the unequal weights

Assumes. A rule of thumb: it assumes equal variances across weighting groups and ignores any gain from stratification (Zipf and Valliant).

With one weight per stratum: n_eff = (Σ n_h w_h)² ÷ Σ n_h w_h², in Excel =SUMPRODUCT(C2:C5,D2:D5)^2/SUMPRODUCT(C2:C5,D2:D5^2).

Mean squared error

\text{MSE} = \text{bias}^2 + \text{variance}

$\text{bias}$: The average amount by which the estimate misses the true value, over repetitions of the surveyunits of the estimate
$\text{variance}$: The spread of the estimate over repetitionsunits squared
$\text{MSE}$: The average squared distance between the estimate and the truthunits squared

Assumes. The margin of error reflects the sampling part of the variance only.

Worked examples

A first example: Saltreed Festival’s response rates

Fictional figures

Saltreed Festival (fictional) e-mailed its post-event survey to 2,500 ticket buyers. Final dispositions: 610 completes, 90 partials, 45 refusals and break-offs, 105 bounced invitations, 1,640 with no reply, and 10 addresses that turned out to belong to the festival’s own staff, who are not in the population. Calculate RR1 and RR2.

Remove the ineligible cases: 2,500 − 10 = 2,490 eligible cases. Ticket buyers who didn’t reply or whose e-mail bounced stay in: they are in the population, just not reached or not willing.
RR1 counts completes only: $610 \div 2{,}490 = 0.2450$ , 24.5%.
RR2 counts partials too: $(610 + 90) \div 2{,}490 = 0.2811$ , 28.1%.
Check what the rates don’t say: whether the 1,880 non-completers differ from the 610 who completed in what the survey asked.

Answer. RR1 = 24.5%; RR2 = 28.1%.

What it means. Report both, with the definitions of complete and partial. Dividing by all 2,500, or dropping the bounced invitations from the denominator, would give different numbers that look comparable and aren’t.

A harder example: Weighting Tamberlake’s renewal survey

Fictional figures

Tamberlake Gyms (fictional) surveyed members about renewing. Its 8,000 members are 2,400 off-peak, 4,800 standard and 800 premium; of these, 90, 210 and 150 replied, and 45, 147 and 135 said they would renew. Premium members, who use the app most, replied far more often. Estimate the share of all members who will say they would renew, and say what the weighting costs and what it can’t fix.

Membership	Members (N_h)	Respondents (n_h)	Would renew	Share	Weight N_h ÷ n_h
Off-peak	2,400	90	45	50.0%	26.67
Standard	4,800	210	147	70.0%	22.86
Premium	800	150	135	90.0%	5.33

Unweighted: $(45 + 147 + 135) \div 450 = 0.7267$ , 72.7%. Premium members are 10% of members but a third of respondents.
Weights: $2{,}400 \div 90 = 26.67$ , $4{,}800 \div 210 = 22.86$ , $800 \div 150 = 5.33$ .
Weighted share: $(2{,}400 \times 0.50 + 4{,}800 \times 0.70 + 800 \times 0.90) \div 8{,}000 = 5{,}280 \div 8{,}000 = 0.660$ , 66.0%: 6.7 points lower.
Cost: $\sum w_i = 8{,}000$ (the weights add back to the members); $\sum w_i^2 = 90 \times 26.667^2 + 210 \times 22.857^2 + 150 \times 5.333^2 = 177{,}981$ ; $n_{\text{eff}} = 8{,}000^2 \div 177{,}981 = 359.6$ , a design effect of 1.25.
Limits: the weights fix the mix of membership types. If, within each type, members who plan to leave were less likely to reply, the 66.0% is still too high, and nothing in these data can show it. A follow-up phone call to a random sample of non-respondents would.

Answer. About 66.0% of members say they would renew (weighted), not 72.7%; the weighting reduces the effective sample from 450 to about 360.

What it means. The unweighted figure reflected who was keen to reply as much as who would renew. Weighting removed the part of that bias that runs through membership type, at the price of about a fifth of the sample’s precision.

Applied: a business decision: How far can Ana trust the February NPS?

Fictional figures

From 516 completes, the unweighted Net Promoter Score is +27.5. Assess the survey’s errors and decide what Ana should report.

Coverage: the frame is Lantern’s list, so the results describe Lantern’s contacts, not all coffee buyers, and they miss customers who never joined the list.
Sampling and response: 25.7% RR1 (28.5% with partials); 516 completes against 601 planned, so ±4.3 points for a share near 50%.
Non-response by stratum: RR1 of 51%, 31%, 54%, 21%. The strata with the highest response also have the highest scores (+62, +53, +41, +7), so the unweighted score is tilted upwards.
Weight to the list: weights 5.84, 9.71, 5.56, 14.58; weighted promoters 40.5%, detractors 20.1%, NPS +20.4. Kish’s n_eff = 469.
Remaining risk: within each segment, the keenest customers may have been the likeliest to reply. Nothing in the survey can rule that out, so the weighted score is better described as an upper estimate than as the truth.

Answer. Report the weighted score, +20.4, with the response rate (25.7%), the weighting to the list’s four segments, and a plain warning that customers who like Lantern may have been keener to reply, so the true score could be lower.

What it means. The difference between +27.5 and +20.4 is the part of the error Ana could see and correct. The rest needs evidence from outside the respondents, such as a follow-up of non-respondents or the order records of those who did and didn’t reply.

Explore

Common mistakes

“Our 26% response rate means the results are unreliable; at 60% they would be fine.”
A response rate measures room for bias, not bias. AAPOR states that response rates alone are insufficient to determine non-response error, and Groves and Peytcheva’s 59 studies found little correlation between the two. What matters is whether non-respondents differ on what you are measuring.
“Weighting removes non-response bias.”
Weighting corrects the mix of respondents on the weighting variables only. Lantern’s weighted NPS, +20.4, still overstates the truth because within every segment promoters replied more readily. Pew found that even the best adjustments left most of the bias in opt-in samples.
“The margin of error is the error.”
It reflects sampling variance alone. Coverage gaps, non-response, wording, mode and processing mistakes can each be larger, and none of them shrinks as the sample grows.
“Response rate = replies ÷ invitations, however you count.”
Whether partials count (RR2) or not (RR1), and what happens to bounced invitations and ineligible cases, change the figure. Use AAPOR’s definitions, set your complete and partial rules in advance, and say which rate you report.
“Weights are free.”
Unequal weights reduce precision: Lantern’s 516 completes carry about the information of 469 equally weighted ones by Kish’s rule. Very large weights on a few respondents are a warning sign; collapse small groups or trim extreme weights, and say so.
“38% said they would buy, so 38% will buy.”
Stated intentions overstate purchases, especially for new products (Sun and Morwitz), and predict less well for new, non-durable products and long horizons (Morwitz, Steckel and Gupta). Treat intent as a signal of interest and apply an explicit, tested conversion assumption when sizing a market.

Guided practice

Work through these with hints when you need them. Every hint and every worked solution is there to be used.

Loading four questions…

On your own

No hints this time. Answer, check, and read the feedback; the solution appears once you have answered or if you ask for it. Your first answers count toward mastery.

Loading six questions…

In practice

Professional surveys publish their quality, not just their results. AAPOR’s definitions exist so that a reader can compare one survey’s response rate with another’s; many statistics offices publish quality reports listing response rates, weighting variables and known coverage gaps; and good market research reports carry a method statement with the frame, the dates, the mode, the dispositions, the weighting and the margin, so that a reader can judge the errors the numbers can’t show.

The most useful habit is a non-response analysis. Before reporting, compare respondents with non-respondents on everything the frame records. A retailer surveying loyalty-card holders can compare the two groups’ spending, visit frequency and tenure from its own transaction data; if respondents spend 40% more, every satisfaction figure is suspect, and the transaction variables become candidates for weighting. When a decision is large, budget for a follow-up of a random sample of non-respondents by a different mode.

Fictional case

Ana’s survey note to herself, filed with the results: “Invited 2,004; RR1 25.7%, RR2 28.5%; response ranged from 21% (newsletter only) to 54% (workshop participants). Results weighted to the list’s four segments; effective sample 469. Weighted NPS +20.4, probably an upper estimate because keener customers are likelier to reply. Cold Brew intent is interest, not a forecast. Next time: compare respondents’ and non-respondents’ order histories, weight on them if they differ, and phone a random 50 non-respondents.”

None of this makes a survey worthless. It makes its claims match its evidence. A weighted score with an honest warning, a response rate stated by a standard definition and an intent figure treated as interest will survive scrutiny; a raw +27.5 presented as “what our customers think” will not.

Knowledge check

Six questions, one attempt each, no hints. Score 80% or more to complete the lesson; you can take it again at any time.

Loading six questions…

Summary

Total survey error: coverage, sampling, non-response and adjustment errors on the representation side; validity, measurement and processing errors on the measurement side. The margin of error measures only sampling variance; MSE = bias² + variance.
AAPOR’s final dispositions: completes and partials; refusals and break-offs, non-contacts and others; unknown eligibility; not eligible (excluded). RR1 = I ÷ all eligible and unknown cases; RR2 adds partials to the numerator. Say which rate you report.
Non-response bias = (1 − RR) × (respondents − non-respondents). A low rate allows bias; it doesn’t prove it, and a high rate doesn’t prevent it. Bias differs from question to question.
Post-stratification weights w_h = N_h ÷ n_h make the respondents match the population on the weighting variables; the weighted estimate equals the stratified estimate.
Weighting costs precision: Kish’s n_eff = (Σw)² ÷ Σw², deff = 1 + CV²(w), a rule of thumb. It can’t fix differences between respondents and non-respondents within a weighting group.
Stated purchase intentions tend to overstate buying, especially for new products, and relate less closely to purchases for new products, non-durables, long horizons and whole categories. Treat them as interest, not forecasts.
Processing errors (coding, merging, keying, weighting the wrong rows) are prevented by a data plan, checks against the frame and second coders.

Key formulas

AAPOR response rates RR1 and RR2
Non-response bias of a respondent mean or share
Post-stratification weights and a weighted estimate
Kish effective sample size and the design effect of weighting
Mean squared error

Terms introduced

Total survey error: The combined error in a survey estimate from every source (coverage, sampling, non-response, adjustment, measurement and processing), considered together when designing and judging a survey.
Coverage error: Error in a survey estimate that arises because the sampling frame misses part of the target population (undercoverage) or includes units outside it (overcoverage), and those units differ on what is measured.
Measurement error: The difference between the answer a respondent gives and the true value, from wording, response styles, order, mode or memory. Its systematic part is measurement bias.
Processing error: Error introduced after answers are collected, in recording, coding, editing, merging or weighting the data.
Mean squared error: The average squared distance between an estimate and the true value over repetitions of a survey: bias squared plus variance.
Final disposition: The outcome code given to every case in a survey sample at the end of fieldwork, in AAPOR’s four groups: interview (complete or partial), eligible non-interview, unknown eligibility, and not eligible.
Break-off: A respondent who starts a questionnaire but stops before answering enough questions to count as a partial; AAPOR counts break-offs with refusals.
RR1: AAPOR’s Response Rate 1: complete interviews divided by all eligible cases and cases of unknown eligibility, I ÷ ((I + P) + (R + NC + O) + U).
RR2: AAPOR’s Response Rate 2: complete and partial interviews divided by all eligible cases and cases of unknown eligibility, (I + P) ÷ ((I + P) + (R + NC + O) + U).
Unit non-response: A sampled person or unit giving no usable answers at all, so that it is missing from the respondents.
Item non-response: A respondent leaving particular questions unanswered while answering the rest of the survey.
Survey weight: The number of population members a respondent stands for in an estimate, used so that weighted respondents reproduce the population’s make-up.
Post-stratification: Weighting respondents after a survey so that each group’s weighted total matches its known population count: each respondent in group h gets the weight N_h ÷ n_h.

Sources

Groves, R. M. & Lyberg, L. (2010). “Total Survey Error: Past, Present, and Future”. Public Opinion Quarterly 74(5), 849–879. doi:10.1093/poq/nfq065 (listed in the issue’s contents). academic.oup.com/poq/issue/74/5
American Association for Public Opinion Research (2023). Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys, 10th edition. aapor.org/wp-content/uploads/2023/05/Standards-Definitions-1…
National Research Council (2013). Nonresponse in Social Science Surveys: A Research Agenda, chapter 2 “Nonresponse Bias”. The National Academies Press. nationalacademies.org/read/18293/chapter/4
Zipf, G. & Valliant, R. (2025). “Design Effects and Effective Sample Size”. Vignette for the R package PracTools, CRAN. cran.r-project.org/web/packages/PracTools/vignettes/Design-e…
Mercer, A., Lau, A. & Kennedy, C. (2018). “For Weighting Online Opt-In Samples, What Matters Most?” Pew Research Center. pewresearch.org/methods/2018/01/26/for-weighting-online-opt-…
Morwitz, V. G., Steckel, J. H. & Gupta, A. (2007). “When do purchase intentions predict sales?” International Journal of Forecasting 23(3), 347–364. academia.edu/35359803/When_do_purchase_intentions_predict_sa…
Sun, B. & Morwitz, V. G. (2010). “Stated intentions and purchase behavior: A unified model”. International Journal of Research in Marketing 27(4), 356–366. english.ckgsb.edu.cn/wp-content/uploads/2017/12/intention.pd…

Before you start

You will be able to

Review first

The idea

Total survey error

Response rates by the book

Why a response rate is not a measure of bias

In depth

Weighting to the population

What weighting costs

What weighting can’t fix

Measurement error: wording, mode and memory

Stated intentions are not purchases

Processing error

Formulas

AAPOR response rates RR1 and RR2

Non-response bias of a respondent mean or share

Post-stratification weights and a weighted estimate

Kish effective sample size and the design effect of weighting

Mean squared error

Worked examples

Explore

Common mistakes

“Our 26% response rate means the results are unreliable; at 60% they would be fine.”

“Weighting removes non-response bias.”

“The margin of error is the error.”

“Response rate = replies ÷ invitations, however you count.”

“Weights are free.”

“38% said they would buy, so 38% will buy.”

Guided practice

On your own

In practice

Knowledge check

Summary

Key formulas

Terms introduced

Sources