Research Bias in User Research: How It Creeps In and How to Curb It
Bias distorts what users say and what the team concludes. Learn the most common biases, how to spot them, and the practical steps that reduce them.
Why is it so easy to fall into bias?
Because research with people is a conversation, and conversations are full of signals. A smile at the right moment, a question that starts with “don’t you think…?”, or a task explained with too much detail can change the answer without anyone noticing. The team also arrives with hypotheses, which makes it read the data in favor of what it already believes.
Nielsen warned about this years ago: if you ask leading questions or press respondents for answers, they may make up opinions that do not reflect their real preferences (NN/g: Interviewing users). In other words, bias does not only distort the data; it can create false data that looks true.
Which biases are most common?
UX Magazine lists ten types of bias. These are some of the ones that show up most often in everyday studies (UX Magazine: 10 types of cognitive bias in UX research):
| Bias | Where it happens | Typical example |
|---|---|---|
| Sampling | Recruitment | Recruiting only acquaintances or very similar people |
| Interviewer | Conversation | Asking “the button frustrated you, right?” |
| Confirmation | Analysis | Highlighting the notes that support the hypothesis |
| Social desirability | Response | Saying “I always read the terms” when you don’t |
| Recency | Analysis | Giving more weight to the last two interviews |
| Hawthorne effect | Observation | Changing behavior because people know they are watched |
| Sponsor | Study design | Looking for findings that justify what the boss wants |
The same list includes false-consensus bias (assuming others think like us), the curse of knowledge (forgetting that the user does not know what we know), and design bias.
What are leading questions?
They are the most direct bias and the easiest to avoid. NN/g defines them as questions that include or imply the desired answer. For example, “What did you find easy about finding the cart?” already says it was easy. The neutral version is “What happened when you looked for the cart?”
NN/g recommends not rephrasing the participant’s words, not suggesting answers, not naming an interface element before the person mentions it, and not assuming how they feel. Open questions such as “What was easy or difficult about…?” help. And review the script with a colleague before the first session: almost everyone asks leading questions without meaning to, and they are hard to correct on the fly.
What can you do about social desirability?
People tend to say what they think we want to hear. Measures that help:
- One-on-one interviews rather than group sessions, because in a group it is easier to adjust your answer to what others will see as favorable.
- Explicit confidentiality at the start: UX Magazine recommends reinforcing it in studies.
- Questions about specific, recent situations: “What did you do the last time you…?” rather than “How do you usually do it?” Nielsen warns that asking how someone “usually” does a task produces an idealized workflow, not the real one.
- Critical-incident questions: ask for one specific, vivid case.
What about moderator bias?
The moderator is part of the problem. Tone, gestures, and task order all shape the answer. UX Magazine recommends that the interviewer’s facial expressions stay as neutral as possible and, to counter the curse of knowledge, that you speak to participants as if you were also new to the topic. Another common practice is that the person who designed the solution does not moderate its test. If you are the designer, ask someone else to run the test of your work, or at least record the session so you can review it later.
How do you reduce bias in analysis?
Confirmation bias appears when the team reviews notes and highlights only what supports its idea. Three practices help:
- Review all the data, not only the latest sessions, to counter recency bias.
- Look for counterexamples. Ask what you observed that does not fit the hypothesis.
- Triangulate. Combine interviews with observation, analytics, or surveys. Nielsen notes that triangulating guards against misleading outcomes.
To turn notes into themes without favoring one idea, the research synthesis guide offers a step-by-step process.
When is bias not the main problem?
- If the study is exploratory and only looking for ideas, bias matters less than the lack of good questions. Moderate bias in an open conversation rarely changes its direction.
- If the sample is very small, fixing bias is not enough. Check the sample size first.
- If the decision depends on a metric, the important question is what to measure, not what someone said. See NPS, CSAT, and CES to choose the right metric.
Illustrative example
Illustrative example: a team wants to know whether users understand a new “Save draft” button. The first script asks, “How clear is the draft button?” Three of five people say it is clear. Reviewing the script with a colleague reveals the trap: the question assumes the button exists and matters. The new script starts with “What would you do if you wanted to finish this later?” and observes where the person looks. Result: only one of five finds the button without help. With the first question, the team would not have found the problem: the question itself handed the participant the answer.
Before you start, build a research plan with neutral questions. That removes most biases before they can show up in the conversation.
Common mistakes
- Asking what you want to confirm instead of what you want to discover.
- Asking yes-or-no questions when you are looking for understanding.
- Always recruiting the same people and calling that group “users.”
- Reading only the notes that support the team’s hypothesis.
- Ignoring social desirability because the person seemed sincere.
- Letting the person who designed the solution also test and interpret it.
What you can do tomorrow
- Read your interview script and mark every question that contains an implied answer. Rewrite each one as an open question.
- Ask a colleague to moderate a short test of your design, or record one of your own sessions and count how often you talked too much.
- Before analysis, write in one column the observations that contradict your hypothesis.