AI-Moderated Usability Tests: What You Gain and What You Lose

An AI-moderated test uses an agent to guide a usability session. A CHI 2026 trial compared AI and human moderators. When it works and when it does not.

info Quick Definition
An AI-moderated usability test is a session in which an AI agent guides the participant: it presents the task, asks probing questions, and asks the person to think aloud. The moderator is not a person, but the test still needs design, judgment, and review.

What does an AI moderator do?

In a traditional moderated test, a researcher reads the task, watches the screen, interrupts when needed, and asks follow-up questions. An AI moderator covers some of those functions: it gives the instructions, asks “what did you expect to happen?” when someone stops, and records what the person says. Great Question’s guide describes these interviews as qualitative sessions in which an AI agent conducts the conversation end to end and decides on follow-ups based on what the participant says.

The key difference is what the agent observes. If it only listens, it misses what the user does on screen. If the platform records clicks, timings, and navigation, the analysis looks more like a traditional test.

What does the evidence say?

A randomized controlled trial published at CHI 2026 by a team from the Hong Kong Polytechnic University compared an AI audio moderator with a human one in think-aloud usability sessions. It included 60 participants and used a note-taking app. Before the trial, the team interviewed nine UX experts to design the moderator. According to the authors, there were no significant differences in task performance or in verbalization. AI moderators, however, received lower social perception ratings.

That result has two useful readings for a designer. First, on a bounded task like using a note-taking app, an AI moderator can carry the session. Second, the participant’s experience changes, and that matters if the session touches a sensitive topic or if you need the participant to trust the person listening. It is a single study with one app and a small sample: do not treat it as a general rule.

Earlier work was more cautious:

  • In his December 2024 roundup on UX Tigers, Jakob Nielsen discusses a study by Kurica, Demcak, and Krajcovic with 60 participants on an e-commerce site. After each task, GPT-4 generated one to three follow-up questions. Tasks ran in a fixed order and the AI could not see how people interacted with the prototype: it analyzed only their comments. There was no significant difference in the number of usability problems found, and the AI’s questions added some extra feedback but rarely deeper insight.
  • A thesis from the University of Gothenburg tested GPT-4o as a moderator under the relaxed think-aloud protocol, in nine sessions on an e-commerce site. Through non-leading questions, the model kept participants verbalizing across the three levels analyzed, but it sometimes slipped into directive or overly affirming language.

When does it make sense?

  • When you have many sessions and little time to moderate them all.
  • When the task is well defined and the questions follow a clear script.
  • When the topic carries low emotional risk: purchasing, changing settings, navigation.
  • When you want a first round to find coarse problems before a human-moderated test. Great Question suggests exactly this hybrid approach: AI for breadth, a researcher where nuance matters.

How do you validate it before trusting it?

A short validation flow:

  1. Run two pilot sessions on the same task, one with a human moderator and one with the AI.
  2. Compare three things: whether people complete the task, how many comments they give, and how they describe the experience.
  3. Review the transcripts for moderator replies that hint at the solution, because they contaminate the finding.
  4. Watch the screen recordings, not only the audio.
  5. Decide the scope: which tasks the AI moderator runs and which stay human.

A common mistake is validating only with the transcript. If the person clicked the wrong button and did not say so, the transcript does not record it.

When should you NOT use it?

  • For health, finance, grief, or any sensitive emotional topic. Great Question’s guide says sensitive topics, high-stakes conversations, and genuinely exploratory research still benefit from a human.
  • For new flows where you do not yet know what to ask: a human moderator improvises better.
  • For accessibility studies with participants who use screen readers or alternative communication, unless you have validated the platform with those people.
  • When the goal is to observe nonverbal language or silent frustration.

Example

Illustrative example: a team wants to test a new password-change flow with forty people. It uses an AI moderator with screen recording and a written task. It finds that 30 percent of participants stop at the same step, without the moderator asking for help. With that signal, the team runs six human-moderated sessions and confirms the problem is the error message text, not the step itself.

What it learned: the AI helped it find where to look. The explanation came from the human moderator.

How do you write the AI moderator’s instructions?

The moderator does what its instructions allow. A moderation prompt should include:

  • The goal of the test, not the solution: “understand whether the person finds the export button,” not “test the new button.”
  • The task, written the way a real person would phrase it: “They want to send the report to their manager. What would they do?”
  • Intervention rules: when it may ask, how many times, and with which phrases. For example, after 60 seconds without progress, it may ask “what are you thinking?”, but it must never point to where the button is.
  • What it must not say: judgments about the answer, praise that confirms a hypothesis (“great, you found it”), and hints about the interface. The Gothenburg thesis shows why: the model sometimes turned directive or overly affirming.
  • A closing: one general experience question at the end, and a neutral thank-you.

Test the instructions in a session of your own before the first real test. If the moderator helps too much in that session, adjust the rules before you spend participants.

Common mistakes

  • Using an AI moderator for sensitive topics without validating it.
  • Validating only with transcripts, without seeing what the person did.
  • Letting the moderator suggest the solution in its probing questions.
  • Reporting the sample size without saying how many people completed the task.
  • Confusing “more sessions” with “better evidence” when the questions are biased.
  • Generalizing the CHI 2026 result (one note-taking app, 60 people) to any product.

Sources

Frequently asked questions

What is an AI-moderated test?
It is a usability test where an AI agent guides the session: it gives the task, asks probing questions, and asks the person to think aloud. In the CHI 2026 study the moderator was audio-based.
Are AI-moderated test results the same as with a human moderator?
A CHI 2026 randomized trial with 60 participants in a note-taking app found no significant differences in task performance or verbalization. It did find that AI moderators received lower social perception ratings.
What do I lose with an AI moderator?
According to Great Question’s guide, AI has limited ability to read what the person does not say, and sensitive topics still benefit from a human moderator. If the platform records only transcripts, you also miss the behavior on screen.
When is an AI-moderated test a good idea?
For studies with many sessions, well-defined tasks, and topics with low emotional risk. For sensitive topics or new flows, a human moderator is still the better choice.