Qualitative research

In depth

Recruiting the right people

Qualitative participants are chosen on purpose, not at random: the study needs the kinds of people whose views matter to the decision. Lantern needs two kinds, its own subscribers and the much larger group of ready-to-drink buyers who have never bought from it, and puts each in its own group so that neither shapes the other’s discussion. Ana can invite subscribers from her list; the second group needs an agency, which finds people with a screener: a short questionnaire that decides whether a person qualifies for a study, and for which group, before they are invited.

The screener for group B (fictional) shows what a screener does:

1. Age: 18 or over (adults only, so no parental consent is needed).
2. Bought ready-to-drink coffee at least once in the past month: must say yes.
3. Which of these coffee brands have you bought in the past year? (Lantern appears in a list of eight, so the screener doesn’t reveal who is asking.) Anyone who has bought Lantern is excluded.
4. Works, or has a close family member who works, in marketing, market research or the coffee or drinks trade: excluded.
5. Took part in a group discussion or paid interview in the past six months: excluded, to avoid people who take part for a living.
6. Where they usually buy ready-to-drink coffee (supermarket, convenience store or petrol station, specialty grocer): the agency recruits a mix.

Recruits are told at recruitment, and again at the start of the session, that it will be recorded and that Lantern’s team may watch; the MRS Code of Conduct (rule 29) requires participants to be informed about any recording, monitoring or observation at recruitment and at the beginning of data collection. They are paid a thank-you for their time, and the agency recruits one or two more than needed, because some people won’t come.

The discussion guide

A discussion guide is the moderator’s plan for an interview or group: the topics in order, the main questions, any tasks, and the time for each. Most guides follow a funnel, from the general to the specific, so that participants describe their own habits and words before they see anything the client has made. In Lantern’s guide the can appears only after 40 minutes, once everyone has described how and where they buy coffee and which canned brands they know.

The questions are open (“tell me about…”, “how do you decide…”) and the follow-ups are probes: neutral prompts that ask for more without suggesting an answer, such as “What makes you say that?”, “Can you give me an example?” and “What else?” A leading question (“Wouldn’t a stylish can make you more likely to buy it?”) tells participants what the client hopes to hear, and in a group one agreeable answer can carry the room.

How many interviews or groups?

Qualitative samples are sized by saturation: the point at which further interviews or groups add no new themes. Two studies give it numbers. Guest, Bunce and Johnson (2006) coded 60 in-depth interviews with women in Ghana and Nigeria in rounds of six and found that saturation occurred within the first twelve interviews, with the basic elements of the main themes present as early as six. They warned that twelve will likely not be enough if the group is relatively heterogeneous or the topic diffuse.

Hennink and Kaiser (2022) reviewed 23 articles that tested saturation, 17 of them with empirical data, and found that those studies reached it within 9 to 17 interviews or 4 to 8 focus group discussions, mainly where the participants were similar to one another and the objectives narrow. The practical rule follows: plan enough interviews or groups for each kind of person, watch how many new codes each one adds, and stop when a run of them adds nothing.

By either standard, Lantern’s design is thin: one group of each kind can’t show that the themes are complete. Its findings are a first map, good for wording the next survey’s questions and for hypotheses, not a full account of what ready-to-drink buyers think. M01’s 12 catering-client interviews match the twelve within which Guest and colleagues found saturation in their fairly similar group, but they have a different weakness: every interviewee was a current client.

From transcripts to themes

Analysis starts from transcripts, word-for-word written records of each recording. Coding labels each passage with a code, a short name for what it is about, so that passages on the same idea can be collected and compared. A codebook lists the codes with a definition, a rule for when to use each and when not, and an example, so that different coders, or the same coder on different days, code alike. Lantern’s codebook has six themes:

The method most researchers follow is thematic analysis, set out by Braun and Clarke (2006) in six phases: familiarising yourself with the data, generating initial codes, searching for themes, reviewing themes, defining and naming themes, and producing the report. The phases loop: reviewing the themes often sends the analyst back to the transcripts, and the codebook changes as the analysis develops. A theme is a pattern of meaning across the data, not just a heading; “price” becomes a theme when the analyst can say what people mean by it, for example that paying a dollar more than for the brand they buy now needs a reason.

Counting codes honestly

Counting codes shows where the talk went. In Lantern’s 20 coded quotations, 4 are about price and 4 about where to buy, and 4 different participants raised price. But the 20 are passages Ana selected and coded from three hours of talk, so counts describe her selection, not a population. The moderator’s guide decided how long each topic was discussed, talkative participants supply more quotations, and the 15 participants were chosen, not sampled, so “4 of 20 quotations” is not “20% of buyers”.

Report counts, if at all, as what they are (“price came up in both groups, raised by four participants”), and keep percentages for descriptive research. The interesting differences are often between groups: in Lantern’s, subscribers talked about matching Lantern’s coffee and its compostable bags, while the non-customers talked about noticing an unknown brand on a shelf and paying more than they pay now.

Do two coders agree?

Inter-coder agreement is the extent to which two people coding the same material independently assign the same codes. The simplest measure is percent agreement: Ana and a freelance researcher, coding separately, agreed on 18 of the 20 quotations, 90%. But two coders would agree on some quotations by chance, and more often when one code dominates.

Cohen’s kappa, introduced by Jacob Cohen in 1960, corrects for that: $\kappa = (p_o - p_e) \div (1 - p_e)$, where $p_o$ is the observed agreement and $p_e$ the agreement expected by chance from how often each coder used each code. The second coder put B4’s remark about a free can under Price and A7’s remark about compostable bags under Can and label (the case records only that the coders agreed on 18 of 20; which two they split on is this lesson’s assumption); with those, $p_e = 69 \div 400 \approx 0.17$ and $\kappa = (0.90 - 0.1725) \div (1 - 0.1725) \approx 0.88$. McHugh (2012) argues that a kappa below 0.60 shows inadequate agreement.

The disagreements matter more than the score. Each is a sign that the codebook is unclear, so the coders discuss them and add a rule: a free can offered so people can taste it belongs under Taste, and a remark about what the can is made of belongs under Sustainability. Both quotations stay where Ana put them, and the codebook above now says why.

Reporting what people said

• Choose quotations that represent a theme, and say how widely it was shared and by whom (“raised in both groups”).
• Include the views that cut against the main pattern; a report of only the agreeable quotations misleads.
• Keep the words as spoken; mark any cut with an ellipsis and never join remarks from different people.
• Identify participants only by codes such as A3 or B2, and leave out any detail that could identify them.
• State the design next to the findings: who took part, how they were recruited and what the method can’t show.

These are the codes’ honest-reporting rules applied to words: the ICC/ESOMAR code requires that published results are not misleading, and the MRS code that findings are clearly supported by the data.