AI Survey Analysis: Open Answers, Coding, and Limits

What part of a survey does AI analyze?

Surveys have two kinds of data. Closed answers, such as a scale from 1 to 5 or a multiple choice, already come coded and need no AI; you count them in a spreadsheet. Open answers, such as “why did you answer that way?”, are text and need reading. That is where AI helps.

Coding means assigning each answer to a category from a list defined beforehand, the codeframe. The typical case: two hundred open answers about why a user abandoned a flow. Reading them all takes hours. A model can assign categories in minutes, but the result must be reviewed. For interviews, which are synthesized differently, see AI interview synthesis and AI affinity mapping.

What does the evidence say?

A study by von der Heyde, Haensch, Weiß and Daikeler (an arXiv preprint from June 2025, accepted at Survey Research Methods) tested this on a random sample of 5,072 answers from the GESIS panel, in German, to a question about why people take part in the panel. Two human researchers coded them independently with 22 categories (Cohen’s kappa of 0.91 between them; kappa measures how much two coders agree beyond chance). The authors compared GPT-4o, Llama 3.2 3B and Mistral NeMo, with instructions only, with examples, and with fine-tuning.

What they found:

• Performance varies a lot between models. GPT-4o was the best, with a macro F1 score (which averages the accuracy of all categories; 1 is perfect) of around 0.7 with the three approaches that did not use fine-tuning. Only the fine-tuned GPT-4o reached a level the authors call satisfactory: 0.87.
• Frequent categories come out well. For incentive, interest and fun, F1 was around 0.9.
• Catch-all categories come out badly. For “no reason” (which includes “don’t know”, fillers such as “???” and blank answers), “non-identifiable/other” and other survey characteristics, F1 was between 0.02 and 0.3. The models often treated those answers as if they contained no response: with GPT-4o, more than 60% of the “no reason” answers were left unclassified.
• The final distribution changes. Because each category is hit differently, the percentage breakdown from a model without fine-tuning looks little like the human one.

Some limits deserve a careful read. It is one language, one question and 2024 models; today’s models may do better, so treat it as a warning about what to watch, not as a number for your case. And fine-tuning, the approach that worked, needs a large set of human-coded answers and more resources, so the authors conclude that it is not yet a cheap alternative. They also warn that setting the temperature to zero and fixing a seed did not guarantee identical results in every model.

Another arXiv preprint (December 2025) proposes repeating a thematic analysis with several models and using Cohen’s kappa to measure how much the passes agree with each other. In its proof of concept, with a single interview transcript and six passes per model, it obtained kappas between 0.84 and 0.91. It helps detect instability, but agreeing with itself is not the same as being right: you still need a sample coded by people.

How do you apply it with validation?

A flow for a survey with hundreds of responses:

1. Define the codeframe before using AI: each category with a definition and an example.
2. Create explicit catch-all categories: “other,” “don’t know,” and “blank,” so the model does not leave them unassigned.
3. Ask for the coding with the codeframe, the definitions and one example per category. Ask for a single category per answer, as in the study.
4. Code a sample by hand (for example, 50 to 100 answers, a rough size) without looking at the AI result.
5. Measure agreement per category, not only overall. A high overall agreement can hide a category that fails.
6. Refine and repeat in the categories that failed. Change the order of the categories in the prompt and check that the counts do not move: the authors of the German study randomized that order in every query to avoid order bias.
7. Report the method, the model and the limitations (see below).

When does AI actually save time?

The effort of preparing the codeframe and validating a sample is almost fixed, and the saving grows with volume. With several hundred answers and clear categories, AI plus a validation sample usually costs less than reading everything. With forty answers, the fixed effort eats the saving, and reading them directly is faster and more reliable.

How do you report survey results coded with AI?

A report that used AI for coding should let another person understand what you did and repeat it. Include:

• The number of valid, blank, and “don’t know” answers, reported separately.
• The complete codeframe, with the definition and one example per category.
• The model, version and date of the coding, and whether it involved fine-tuning or only instructions.
• The validation result: how many answers you checked by hand and the agreement per category.
• Limitations: which categories had low agreement and how you handle them in the conclusions.

Present percentages next to the absolute number: with 40 answers, 25 percent is ten people, and saying only “a quarter” hides how small the base is. If a category has low agreement, do not remove it from the report: mark it as uncertain.

What should you NOT delegate?

• Conclusions about what the results mean for the business.
• Analysis of a survey with very few answers: read them all.
• Sensitive answers or answers from vulnerable people, without human review or a conscious decision about where the data is processed.
• Surveys in languages or contexts where you do not know how the model behaves: validate first.

Example

Illustrative example: a designer receives 300 open answers about why users did not complete a credit application. She defines ten categories plus “other” and “don’t know,” each with a definition and an example. The AI codes everything. She codes 80 answers by hand without looking at the result and finds the model confuses “unclear requirements” with “slow response.” She refines those two definitions, recodes, and agreement in those categories rises. In the report she puts the number of answers and the agreement for each category.

Common mistakes

• Letting the AI invent categories without a prior codeframe.
• Not creating categories for “don’t know” and blank answers, or not checking how the model treated them.
• Trusting the overall agreement without looking at each category.
• Validating with the same person who wrote the prompt, without a second look.
• Reporting percentages per category without the number of answers and the method.
• Using AI for a small survey where reading was faster.

Sources

• von der Heyde, Haensch, Weiß and Daikeler, AIn’t Nothing But a Survey? Using LLMs for Coding German Open-Ended Survey Responses on Survey Motivation (arXiv 2506.14634)
• Jain, Adeyinka, Roseman and Allsop, Multi-LLM Thematic Analysis with Dual Reliability Metrics (arXiv 2512.20352)