Generative vs. evaluative research: when to explore and when to validate
Generative research finds which problem is worth solving; evaluative research checks whether a solution works. Differences, methods and when to use each.
What is each one?
Generative research is for discovery. When the team knows a metric is off but not why, or when it wants to enter a new market, it needs to understand motivations, contexts and needs before proposing anything. According to UserTesting, it typically uses qualitative methods to understand user needs and motivations.
Evaluative research is for checking. When a concept, a sketch, a prototype or a product already exists, the team wants to know whether it works and what should improve. UserTesting describes it as typically using quantitative methods to assess the success or failure of a product or feature.
There is an important nuance: in practice, the line is not that clean. UserTesting itself notes that, in evaluative research, researchers may also use qualitative methods. Usability tests, which are evaluative, can be qualitative. Interviews, which are generative, can include closed questions. What separates them is the question the study tries to answer, not only the method.
Why does it matter to a designer?
Because it avoids two common mistakes. The first is evaluating something nobody needed: a team tests a flawless screen for a feature users never asked for. The second is exploring endlessly when there is already enough material to design and test.
It also helps shape the process. Generative research feeds product decisions and evaluative research validates the design work. A team that knows which phase it is in can say what research it needs without debating it every week.
Which methods does each use?
In the NN/g article by Christian Rohrer, methods are classified by product phase. It does not use the word “evaluative”: it calls the methods of the strategy phase generative, those of the design phase formative (they inform how to improve the design) and those of launch and assessment summative. Evaluative research covers the last two. These are the methods NN/g lists for each phase:
| Phase | Typical methods | Question it answers |
|---|---|---|
| Strategy (generative) | Field studies, diary studies, interviews, surveys, participatory design, concept testing | What problems do people have, and what is their context? |
| Design (formative) | Card sorting, tree testing, usability testing, remote testing | Does this proposal make sense and work? |
| Launch and assessment (summative) | Usability benchmarking, unmoderated testing, A/B testing, analytics, surveys | How does the product perform against earlier versions or competitors? |
NN/g warns that these classifications are general guidelines, not rigid categories, and that concept testing, desirability studies and card sorting have both qualitative and quantitative versions. A card sort, for example, can explore how people think about information or validate a proposed structure.
For the generative side, user interviews and diary studies are the most direct tools. For the evaluative side, usability testing and A/B testing cover most cases.
When should you use each?
The most useful rule is this: use generative research when the problem is unclear, and evaluative research when there is something concrete to test. NN/g places generative methods at the start of a project, to define direction, and formative methods once that direction is chosen.
A pattern that works is to alternate them. You start with interviews to understand a problem, propose three ways to solve it, test the three with users, choose one and measure after launch. Each stage answers a different question, and the previous one feeds the next.
On sample sizes: for a qualitative usability study with a single group of users, NN/g recommends five to uncover most of the common problems. The number changes with the type of study, and quantitative tests are mainly used to collect benchmarks, with a different method and sample. Do not mix both goals in the same study without acknowledging it.
Example
Illustrative example: a trip-planning company sees that half its users drop off when choosing dates. First it runs generative research: interviews with ten people who plan family trips, and a two-week diary with five of them. They discover that the problem is not the dates but the uncertainty about the final price. With that in hand, they propose three prototypes: showing a price range, showing the price per night or showing a colour-coded calendar. Then they run evaluative research on the three prototypes and, after choosing the price-range version, measure drop-off in production with A/B tests.
For your next project: write in one sentence what you do not know yet. If the sentence starts with “why”, you need generative research. If it starts with “whether this works”, you need evaluative research.
When should you NOT use them?
- When the decision is already made and the study only serves to justify it.
- When there is nothing to evaluate. Testing without a prototype, a concept or a product leaves the team without material.
- When evaluative research is used for discovery. A usability test will not tell you which problem is worth solving.
- When the sample does not fit the goal. Five people are not enough to measure a conversion rate.
Common mistakes
- Evaluating before exploring the problem.
- Exploring endlessly when the team already has material to design with.
- Confusing a qualitative sample with a quantitative one, and drawing conclusions from percentages based on five people.
- Believing the method defines the type of research. What defines it is the question.
- Not documenting the phase of the study, which makes later comparison impossible.
- Using A/B tests to decide the direction of the product, when they only compare two concrete options.