Moderated vs. unmoderated testing: when to sit with the user

What does each one offer?

In a moderated test, the facilitator is in control. They can follow up when the participant says something unexpected, ask them to explain what they are thinking and clarify a task they did not understand. NN/g says remote moderated tests are, compared with in-person ones, often less expensive, less time-consuming and more convenient for participants. The challenge in the session is deciding when to speak and when to leave the silence.

In an unmoderated test, the participant is alone with the test. According to NN/g, it fits when the main focus of the study is a few specific elements, or when you want to check the effect of a relatively minor change. It helps on tight timelines, because participants can complete sessions on their own schedules, even at the same time. The cost is depth: follow-up questions are predefined and identical for every participant, nobody helps in real time, and the facilitator does not know how the session went until it ends. Sessions can also be quieter, because nobody reminds the participant to think aloud.

What are the risks of each?

The risk of a moderated test is preparation. NN/g warns that it can require a lot of advance preparation, especially when setting up the tool, and recommends choosing one that is easy for participants to access and install. Mobile-focused tools can be unstable and may not support every iOS or Android version. On top of that comes a known risk of any facilitated session: a badly phrased question can steer the answer.

The risk of an unmoderated test is the quality of participation. NN/g names problem participants as the main danger. Respondent adds that people who know nobody is watching may take the test less seriously, explore the design less or not represent your audience. Spotting and removing those cases is part of the analysis.

How do you write tasks for a test with no facilitator?

Because nobody is there to clarify, wording matters more. NN/g proposes that a task be realistic and actionable, and that it give no clues or describe the steps. Its example: instead of “Purchase a pair of orange Nike running shoes”, use “Buy a pair of shoes for less than $40”, which lets the participant compare using their own criteria. Avoiding clues does not mean being vague: describe the need, not the button. Instead of “Sign up for the newsletter with button X”, write “Find a way to get information about upcoming events sent to your email”.

Before launching, NN/g advises a pilot, a dress rehearsal with one or two people. It shows whether participants understand the tasks, how long the session takes and whether the materials have problems. If it goes well, its data may even be usable in the final results.

How many participants do you need, and what is measured?

NN/g recommends five participants for a typical qualitative study with a single user group, to uncover the majority of the most common problems. Qualitative testing looks for insights and explanations; quantitative testing collects metrics, such as task success or time on task, and is the right one for a benchmark. If you need the second, the study design changes, not just the number of people.

To recruit well, the recruiting participants guide helps make the group representative, something an unmoderated test tests even harder.

When is each one the right choice?

A moderated test fits when:

• the problem is complex and you need to understand the reasoning behind each action;
• the population is hard to reach and each session is valuable;
• you are testing an early concept and want to correct the question on the fly.

An unmoderated test fits when:

• your question is limited to a few elements or the effect of a small change;
• you need quick results and participants are spread out;
• you want each person to take part at their own time, from their own setting.

Many teams combine both: moderated to discover, unmoderated to check. If your test is on a mobile device, also review mobile usability testing.

What if the moderator is an AI?

There is a third route: an AI agent that sets the task, asks probing questions and asks people to think aloud. The evidence is still limited and it has its own costs, explained in AI-moderated usability tests.

Example

Illustrative example: a team wants to know why users of a cooking-class app do not complete enrolment. First it runs five remote moderated tests and discovers that most people misread the “Reserve a place” button, because they think they are already enrolled. The team changes the button text and launches an unmoderated test with the task “Sign up for a pasta class this Saturday”, after a pilot with two people that flagged one confusing instruction. The test checks whether the change fixed the problem, without repeating the long sessions.

For your next study: decide which question you need to answer. If you need to understand why, start with moderated; if you need to check a specific change, consider unmoderated with a pilot first.

Common mistakes

• Launching an unmoderated test without a pilot, with tasks nobody has reviewed.
• Asking leading questions in a moderated test.
• Giving tasks with the steps included, instead of goals.
• Using unmoderated testing to discover unknown problems, when nobody can follow up.
• Underestimating the preparation of a remote moderated test.
• Recruiting people who do not represent the target audience.

Sources

• Remote Usability Tests: Moderated and Unmoderated, Nielsen Norman Group (Amy Schade, 2013)
• Remote Moderated Usability Tests: How to Do Them, Nielsen Norman Group (Kate Moran and Kara Pernice, 2020)
• Remote Usability Testing Study Guide, Nielsen Norman Group
• Usability (User) Testing 101, Nielsen Norman Group (Kate Moran, 2019)
• Turn User Goals into Task Scenarios for Usability Testing, Nielsen Norman Group (Marieke McCloskey, 2014)
• Pilot Testing: Getting It Right (Before) the First Time, Nielsen Norman Group (Amy Schade, 2015)
• Unmoderated Usability Testing 101: How To Get Started, Respondent (Jose Gallegos, 2022)