AI Interview Synthesis: What to Delegate and What to Keep
Interview synthesis turns transcripts into themes and findings. AI helps code and group them, but it loses accuracy as the coding task gets more complex.
What are the steps of synthesis?
Each team organizes it differently, but synthesis has four steps:
- Code: tag excerpts of the transcript with a label.
- Group: bring related labels together into themes. If you work from loose notes instead of transcripts, see AI affinity mapping.
- Contrast: look for cases that do not fit and participants who think differently.
- Conclude: write what each theme means for the product, with its quotes.
AI fits steps 1 and 2 well. Steps 3 and 4 need someone who knows the product and the user’s context. For the interviews themselves, see AI interview scripts; to compare tools that ship with built-in synthesis, see AI research tools.
What does the evidence say about agreement with humans?
A March 2024 report (Kirsten and colleagues, arXiv 2403.06607) compared GPT-3.5 Turbo and GPT-4 with human coders. It used German interview transcripts from a 47-participant study on digital security and privacy. It measured agreement with Cohen’s kappa (1 is perfect agreement) on three tasks with codebooks of different sizes:
| Task (number of codes) | Human vs. human | GPT-4 vs. human | GPT-3.5 vs. human |
|---|---|---|---|
| A: connected devices (18) | 0.97 | 0.94 to 0.97 | 0.74 to 0.85 |
| B: apps and services (24) | 0.83 | 0.74 to 0.75 | 0.56 to 0.60 |
| C: sources of advice (32) | 0.63 | 0.55 to 0.58 | 0.34 to 0.43 |
Three useful readings. On the simplest task, GPT-4 matched the humans. As the codebook and context grew, agreement dropped for everyone, and the gap between humans and the model widened. And adding examples to the prompt did not improve agreement much, but it did reduce formatting errors and invented codes. These are 2024 models: current ones may do better, which is why the number that matters is the one from your own interviews.
A more recent preprint (Jain and colleagues, arXiv 2512.20352) tested Gemini 2.5 Pro, GPT-4o, and Claude 3.5 Sonnet, six runs each, on a single psychedelic art therapy transcript as a proof of concept. Kappa across runs exceeded 0.80 for all three models. Two caveats: that kappa measures how similar the runs are to each other, not whether the analysis is correct, and each model found a different number of themes (6, 5, and 4). A model can be consistent with itself and still miss themes.
How do you apply it with judgment?
A flow that works for small teams:
- Define a short codebook, 10 to 20 codes, with examples of what does and does not belong. The study above suggests that the longer it is, the harder it gets for everyone.
- Ask the AI to code five interviews, with each code and its verbatim quote.
- Check 30 random excerpts against the audio or the original transcript.
- If two people on the team code the same interview, compare; the disagreement shows where the codebook is ambiguous.
- Refine the codebook, and only then code the rest.
- Present the themes with quotes, not percentages without context.
An important detail: the AI output is a draft grouping. Do not bring it to the meeting as if it were the finding. And if you worry about what biases the model introduces, see AI bias in research.
What should you not delegate?
- Deciding which theme matters most to the business.
- Interpreting silences, tone, or contradictions between what the user says and what they do.
- Closing a synthesis without contrasting it with cases that do not fit.
- Presenting a “summary” to stakeholders without verified quotes.
- Replacing interviews with synthetic users. Nielsen Norman Group (Rosala and Moran, June 2024) concludes that they help with desk research and generating hypotheses, but not with making decisions. They tend to please the researcher and overstate positive behavior, and their lists of needs show no priority. For niche groups the warning is stronger.
Example
Illustrative example: a designer has nine interviews about renewing an insurance policy. She asks the AI to code them with a 14-code codebook. She checks 30 excerpts and finds the model confuses “the price is high” with “I don’t understand the price,” two different problems. She refines the codebook, separates the two codes, and codes again. The final themes are three, and each has quotes she verified against the audio.
What she learned: the AI shortened the coding work, but the codebook refinement is what made the result reliable.
When do you stop coding?
There is no fixed number of interviews. A practical criterion, not a scientific rule: keep coding while new interviews add new themes or change the weight of existing ones. When three interviews in a row only confirm what you already have, the synthesis is close to stable. That does not mean the topic is exhausted for every user; it means that, for the group you are studying, you have a stable picture. If an important group is missing from the sample, the synthesis is incomplete even if it has stabilized.
How do you present a synthesis without losing credibility?
A synthesis is judged by how it is presented. A structure that works for product teams:
- One theme per slide or section, with the name written as a claim: “Users do not know when the installment is charged” is better than “Billing.”
- The number of participants who support the theme, without percentages that sound statistical when the sample is small. “Four of nine” is more honest than “44 percent.”
- Two quotes per theme, chosen for clarity, not for how dramatic they are.
- One counterexample: at least one case that does not fit, and what it suggests.
- What it implies for the product and what we still do not know.
That last section adds the most value for whoever decides. A synthesis that only says what happened, without saying what to do or what remains to be tested, leaves stakeholders with the same old question.
Common mistakes
- Accepting the first grouping without reviewing a sample of quotes.
- Using a codebook that is too long or ambiguous.
- Confusing frequency with importance: a theme mentioned by three people can be more serious than one mentioned by ten.
- Letting the AI write the conclusion without the team discussing it.
- Confusing the model’s consistency (it repeats the same result) with the result being correct.
- Not saving the version of the codebook or the model used.