Mobile usability testing: how to run it without losing data
Mobile usability testing evaluates an app on the user's real device. How to recruit, prepare the session, record it and pick tools without losing data.
How is it different from a desktop test?
Raluca Budiu, of NN/g, sums it up: the most important point about a mobile usability study is that it is a usability study, not that it is mobile. The principles are those of any usability test: define tasks, observe, measure where people get stuck and fix it. What changes is the environment, the device and the tools.
Those changes have practical consequences. On a phone the screen is small, users switch apps constantly, the network varies and the operating system and browser version matter a lot. A problem can appear only on Android, only with large text enabled or only with the keyboard in one language.
How do you prepare the test?
Choose participants who use their own device
NN/g recommends recruiting people who are familiar with their device and have been using it for at least three months. The point is to separate difficulty caused by the device from difficulty caused by the app. Someone who just bought a new phone has not yet mastered its gestures, and that contaminates the data. The exception is a study of how hard a new device is to learn.
If your study uses lab devices, bring both types of equipment and note which one you used in each session. Mixing them without recording it is a frequent source of false conclusions.
Run a practice session
For remote moderated tests, Kate Moran and Kara Pernice (NN/g) advise scheduling a 15-minute online meeting with each participant the day before. It checks that screen sharing works, that audio works and that the participant can handle the tool. It is time well spent: a session that drops ten minutes in costs you a participant and an hour.
Look after the environment
NN/g asks participants to be in a quiet, private place with a good internet connection. A café with background noise produces useless transcripts and distracts the participant from the task.
Write tasks with a goal, not steps
A task like “Tap the book button, then the price filter” tests nothing; it tells the user what to do. A task like “You want to book a hotel in Cuenca for Saturday with a budget of 60 dollars a night. Find one” lets the user choose the path, and that is what you need to observe.
How do you record and observe?
On mobile, the phone screen is not always enough. In 2014, Budiu wrote that recording with an external camera was still the preferred method, because it captures the screen and the participant’s fingers at once. Screen-recording apps have improved since, but a screen recording alone still does not show the gesture or the posture, unless the system draws the touches.
The most common combination is this:
- Shared screen or device mirroring to see the interaction in real time.
- Participant video to capture expressions and comments.
- External camera or hand recording, if possible, for the physical interaction.
Limit the number of observers too. According to NN/g, up to two observers can follow the video on a laptop; with more people, project the feed on a separate monitor or projector. Every extra person on a call is someone the participant may feel is judging them, and that can change how they behave.
Which tools should you use?
Desktop remote testing tools do not always work the same way with mobile. NN/g warns that many remote moderated usability testing tools for mobile can have stability issues, and that some do not support all versions of iOS or Android. That is why you should ask participants about their device before recruiting. Check three things: which operating systems the tool supports, which app versions, and whether it can share the phone screen without a complicated installation. To choose between moderated and unmoderated sessions, read moderated vs. unmoderated testing.
If you have no specialized tool, a video call combined with the operating system’s own screen mirroring is enough for a first round. It is less elegant but workable. For the tool landscape, look at Maze and UserTesting, and compare their current options in the official documentation, because they change often.
When is an in-person test better?
Remote testing works well for most flows. There are cases where being in the same room is better:
- When the physical interaction is the problem: a swipe gesture, a photo capture or one-handed use.
- When the context of use matters: an app used on the street, in low light or with hands full.
- When the participant has access needs and requires technical help in the moment.
For accessibility cases, see also mobile accessibility, which works with assistive technologies and not only with usage testing.
Example
Illustrative example: a team wants to test a payments app for a shop. It recruits six people who have used their bank’s app for more than three months, on their own phones. It holds a 15-minute practice session the day before. The tasks are: pay a bill with a QR code, check the latest transaction and split a payment with another person. Two observers follow the session by video, and the screen is mirrored. Three of the six participants cannot find the cancel button because it sits at the bottom, out of reach of their thumb. The team turns this into a finding backed by video evidence, not an opinion.
An exercise for tomorrow: take three tasks in your product and run them on your own phone, one after another. Note where you slow down. If it slows you down, and you know the app, imagine what happens to someone new.
Common mistakes
- Testing on one device and generalizing the result.
- Giving steps in the task, instead of a goal.
- Recruiting people with no experience on the device, and blaming the app for their difficulty.
- Trusting the tool without testing it on the operating system you will use.
- Skipping the practice session and discovering the problem in the first interview.
- Recording only the screen and losing the physical context of the gesture.
- Having too many observers on the session.
- Not recording the model, operating system and app version of each session, and then being unable to explain why a problem only appears for some participants.