Collecting data: sources, samples and bias

In depth

Random error and bias

A sample can miss the truth for two very different reasons. Sampling error is the chance difference between a sample’s result and the population’s that arises because only part of the population was observed. It goes both ways, and it shrinks as the sample grows. Bias is a systematic error that pushes the result in one direction, and a bigger sample doesn’t reduce it at all.

Suppose 30% of a shop’s customers are subscribers, 90% of whom are satisfied, and the other 70% are one-off buyers, 60% of whom are satisfied. The true share satisfied is $0.30 \times 90 + 0.70 \times 60 = 69\%$. A survey that reaches mostly subscribers, say a sample that is 70% subscribers, should expect $0.70 \times 90 + 0.30 \times 60 = 81\%$: 12 percentage points too high.

With 100 respondents, chance alone typically keeps the result within about 8 points of 81%; with 5,000 respondents, within about 1 point. The sample becomes more precise, but around the wrong number. The explorer below lets you try it.

Four biases to look for

Each kind of bias enters at a different step of the path from population to numbers, as the figure above shows.

Sampling bias arises when some members of the population are less likely than others to be included. A survey link placed only in the app leaves out website customers; a poll on a council website misses residents without internet access; phoning during working hours misses people at work. The cure lies in the frame and the selection: a list that covers the population, and a random draw from it. Survivorship and non-response bias are particular forms of the same problem, and many textbooks group all three as selection bias: each changes who ends up in the data.

Non-response bias arises when the people who answer differ, in what is being measured, from those who don’t. The response rate, the share of those invited who answered, tells you how much room there is for it. A low rate doesn’t prove bias, but it means the result could sit almost anywhere.

If 120 of 400 invited hotel guests reply and 90 of them are satisfied, then 75% of respondents are satisfied, but the share of all 400 could be as low as $90 \div 400 = 22.5\%$ (if no non-respondent is satisfied) or as high as $(90 + 280) \div 400 = 92.5\%$ (if all are). People with strong views, good or bad, can be the most likely to reply. Reminders, short questionnaires and a follow-up of a random sample of non-respondents reduce the problem; comparing respondents with non-respondents on what you already know about them, such as channel or order value, helps you detect it.

Survivorship bias arises from studying only the people or things that passed some filter, while those that dropped out are invisible. Ana’s survey could only reach customers who completed an order; the visitors most put off by a bad checkout left no address. The same trap catches a business that studies the satisfaction of its current customers (the dissatisfied have already gone), or that learns what makes products succeed by studying only products still on sale. Ask what the filter was, and who failed it.

Measurement bias arises from the way a value is measured or a question is asked, pushing answers systematically one way. Ana’s survey asked every respondent whether the new checkout was easy to use, including customers who had ordered only through the app, which was not changed. Some of them answered anyway, about a checkout they never saw.

The statement format adds a second problem. Pew Research Center reports research showing that less educated and less informed respondents have a greater tendency to agree with agree/disagree statements, and recommends offering a choice between alternative statements instead. Outside surveys, measurement bias comes from instruments and definitions: scales that read heavy, or a website redesign that changes how visits are counted.

Writing survey questions

Pew Research Center’s guidance on writing survey questions reduces to a few rules. Make every question clear, specific and answerable by every respondent who is asked it, which may mean routing some people past a question with a filter question. Ask about one thing at a time: a double-barrelled question, which asks about two things at once, can’t be answered honestly by someone who likes one and not the other. Offer answer options that are exhaustive and don’t overlap.

Remember that earlier questions colour later ones, and that the order of the options matters too. And pretest: try the questionnaire on a few people and find out what they thought each question meant. Two further rules are standard practice: avoid leading wording that signals the expected answer, and keep rating scales balanced, with as many favourable as unfavourable options.

Experiments and A/B tests, in outline

Transactional data and surveys record what happened while many other things were also happening. September brought a new checkout, but also the end of the summer holidays and whatever else changed that month, so a fall in orders after 1 September can’t by itself prove what caused it. An experiment changes one thing on purpose and compares.

In an A/B test, each visitor is randomly assigned to version A, the existing checkout, or version B, the new one. Version A’s visitors are the control group. Because chance alone decides who sees which version, the two groups are alike on average in everything else (device, day of the week, what they came to buy), so a clear difference in the outcome can be put down to the version, apart from chance. Had Lantern split September’s 8,800 web visits this way, each version would have had about 4,400 visits to compare.

A test is only as good as its plan. Decide the outcome to compare and how long the test will run before starting; assign visitors at random, not by day or by device; run whole weeks, so that weekdays and weekends are both included; change nothing else for one group only; and don’t stop the test the moment the result looks good. Whether a difference is too large to be chance is a statistical question, which B04 Descriptive and Inferential Statistics answers.

Observation, and data from others

Observation finds what nobody thought to measure. Watching five customers try to pay on their phones might have revealed a payment-page fault on mobile browsers within minutes. Five people are not a representative sample, and people behave a little differently when watched, so observation suggests problems rather than measuring how common they are.

Third-party and public data supply what a business can’t collect itself: the payment provider’s records of declined cards, official statistics on household spending, market data bought from research firms. Before using them, read their definitions, coverage and timing, and check the licence: what “a failed payment” means, which customers a statistic covers, how late it is published, and whether you may combine it with your own data.

Consent and privacy, in outline

Collecting data about people brings legal duties. This outline uses the UK GDPR as explained by the Information Commissioner’s Office (ICO), the UK regulator; other jurisdictions have their own rules, so a business checks the law that applies where its customers are. Under the UK GDPR, personal data means any information relating to an identified or identifiable natural person, who can be identified directly or indirectly, for example from a name, an identification number, location data or an online identifier. The ICO notes that pseudonymised data, in which names are replaced by codes such as customer numbers, is still personal data.

A business must have a valid lawful basis to handle personal information. For the UK GDPR the ICO lists seven (consent, contract, legal obligation, vital interests, public task, recognised legitimate interest and legitimate interests) and states that no single basis is better or more important than the others. Recognised legitimate interest was added by the Data (Use and Access) Act 2025, so older sources, and sources on the EU’s GDPR, list six. Where a business relies on consent, it must be freely given, specific, informed and unambiguous, shown by a statement or a clear affirmative action: silence, pre-ticked boxes or inactivity don’t count, and people can withdraw consent easily at any time.

For an analyst, a few habits follow. Collect only what the question needs, and say what it will be used for. Analyse survey answers by customer number rather than by name or e-mail address, and keep the file that links them access-restricted. Never publish results for groups so small that a reader could identify the people in them. None of this is legal advice; when in doubt, ask whoever in the business is responsible for data protection.