Wizard of Oz: testing before building, with a person behind the curtain
In Wizard of Oz testing, a hidden person simulates a system that does not exist yet. Learn when to use it, how to prepare it, and what ethics it requires.
What is Wizard of Oz?
The Usability Body of Knowledge defines it as an evaluation of technology that is not yet implemented, in which an experimenter, the “wizard”, simulates the system’s responses, usually without the user knowing. The name comes from L. Frank Baum’s novel, and is credited to J. F. Kelley, who coined it while working on his dissertation in the early 1980s.
The technique is older than the name. Wikipedia notes that it was already used in human-computer interaction research, and that credit for the term has nuances. What matters is the idea: testing the experience before the technology exists.
Why does it matter to a designer?
Because some ideas are expensive to build while their usefulness is still unclear. An AI recommendation feature, a voice assistant, or a natural-language search can cost weeks of development before you know whether people want them. With Wizard of Oz, you can answer that question with a person writing the responses from another room.
It is also useful for comparing several designs cheaply, and for exploring how users behave when the system responds as if it were real.
How do you prepare it?
The Usability Body of Knowledge proposes a clear sequence:
- Design the simulated interface. It can be a screen, a chat, or a voice surface, as credible as possible.
- Write a test plan with instructions for the facilitator, the wizard, the participants, and support staff.
- Recruit representative users who cover the range of the target population, not whoever is available.
- Design realistic task scenarios, with a goal the participant actually wants to achieve.
- Define how the wizard will respond to each participant action, and train them.
- Tell participants they are using an early prototype whose performance is not yet optimized.
- Run pilot sessions to refine the procedure and give the wizard practice.
- Close each session with a debrief: collect impressions and explain what the wizard was and why it was not disclosed earlier.
Koji suggests a more practical version: define the concrete decision the test should inform, write an operational script for the wizard, and pilot with two participants before starting. A common mistake is running the test without a clear decision at the end.
What are its limits?
Wizard of Oz becomes unreliable in some cases:
- Inconsistent responses. Without precise instructions, the wizard improvises differently in each session. The Usability Body of Knowledge recommends preparing instructions for predictable situations rather than scripting every reaction.
- Speed. If the real system must respond in milliseconds, the wizard cannot simulate that, and the participant notices.
- Slowness and fatigue. Koji notes that a wizard is slower than software, which can confound results. The Usability Body of Knowledge adds that wizard performance can decline during a session because of cognitive fatigue.
- Real errors. According to the Usability Body of Knowledge, the technique does not expose errors caused by system performance or recognition rates unless they are simulated. It is better at revealing problems than at predicting real-world usability.
- Long-term behavior. If what matters is repeated use over weeks, a diary study or a beta program gives better information.
The same source notes that it is hard to evaluate systems with a large graphical interface this way, and that the easiest inputs to simulate are voice and hand movements.
What ethics does it require?
Wizard of Oz involves deception, even a mild one. The Usability Body of Knowledge cites the American Sociological Association (ASA) code of ethics: when deception is part of the research design, false beliefs must be corrected once the research is complete, through a final debriefing that explains why the truth was not disclosed at the start. The same page admits that this standard is not universal in usability testing, but says it deserves to be taken more seriously.
Koji recommends a concrete practice: informed consent beforehand stating that the participant is using a prototype that may not work as advertised, without revealing the human operation during the session, and a debriefing at the end. That is the practice worth adopting on your team.
Example
Illustrative example: an insurance company wants to know whether customers would accept an assistant that answers questions about claims in natural language, before training a model. A designer prepares a chat screen and a script of allowed answers. A team member, the wizard, responds from another room under those rules. After ten sessions, the team sees that customers ask almost always about the status of their claim, not about coverage, and that they get frustrated when the assistant is slow. That tells them what to prioritize before building. At the end of each session, the team explains that the responses were human.
Common mistakes
- Running the test without a concrete decision that depends on the results.
- Letting the wizard improvise without clear rules.
- Not stating in the consent that the product is a prototype.
- Skipping the debriefing, which leaves the participant deceived.
- Using it to measure real response times, something the wizard cannot simulate.
- Skipping the pilot, and discovering problems with real participants.
If your idea is a concept rather than an interaction, concept testing helps you validate the proposal before designing the interaction. For a cheaper version, paper prototyping tests flows without any simulation. And to decide how much detail to show, the prototypes page is a good reference.