011 min
Why UX research exists
Picture a product team at a subscription app. Many people sign up, but few finish onboarding. In planning, the lead designer says the six onboarding screens are too long. The team agrees, cuts them to three and ships the change. Completion barely moves.
The team decided from a guess. Nobody had watched a new user go through onboarding, so nobody knew where people stopped or why. Cutting screens fixed a problem the team imagined and left the real one in place. The time spent building the short version, and then rebuilding it, was lost.
UX research exists to replace that kind of guess with an observation. The reasoning is about cost. A problem found on a paper sketch costs an afternoon to fix. The same problem found after launch costs a rebuild and the users who left in the meantime. The rest of this article follows the onboarding team as they learn what to ask, whom to ask, and how much to trust the answer.
022 min
What UX research studies
UX research looks at three things about the people who use a product: what they need, what they do, and what they believe about it. A need is the outcome a person wants. Behaviour is what they actually do while they try to get it. An attitude is what they say and feel about the product. Each study picks one of these and states it as a research question, such as Where do new users stop during onboarding?
A study has a plan before it has people. The researcher writes the question, picks a method and writes a screener, which is a short set of questions that decides who qualifies to take part. They recruit people who match the team's real users, often with a small payment as thanks, and run the sessions. Then they look for patterns across sessions and report findings with recommended actions, ranked by importance.
Two job titles blur here. A UX designer shapes the product, and a UX researcher studies the people it is for. On small teams one person does both, which makes it easy to forget that the two jobs ask different questions. The term user research is wider for anyone studying users for any purpose. UX research is the part aimed at design decisions. It is also not UI work, which is the visual surface of buttons and screens.
Researchers often turn findings into a shared artefact. A persona is a profile of a typical group of users. A journey map shows the steps one person takes across a task, and where they struggle at each step.
032 min
How researchers sort UX research methods
The onboarding team's next problem is choice. Dozens of methods exist, and a team that picks one by habit gets a habit's answers. Researchers sort methods by the question each one can answer. Christian Rohrer's framework for Nielsen Norman Group draws the main lines.
| Split | One side | Other side |
|---|---|---|
| Attitudinal vs. behavioural | What people say | What people do |
| Qualitative vs. quantitative | Why something happens, and how to fix it | How many people, or how much |
| Generative vs. evaluative | Finding problems worth solving | Judging a design that exists |
| Context of use | Natural use of the product | Scripted tasks set by the researcher |
Attitudinal research records what people believe, and it can only reach what they are aware of and ready to report. Behavioural research watches what happens, so it needs no memory or honesty from the participant. Qualitative methods give a few people's reasons in depth. Quantitative methods give a count across many people, but say little about why. Some methods, such as card sorting, can produce both kinds of data.
A fourth split is about timing. Generative research runs before a design exists and looks for needs and opportunities. Evaluative research runs on a design and asks whether it works.
The splits are guides, not walls. Most projects need several methods, because each one covers what another cannot. The onboarding team has analytics, which is behavioural and quantitative. It tells them where people leave. It cannot tell them why.
041 min
Where UX research fits in a product cycle
The split by timing raises a fair question for the onboarding team: when should they have looked? Susan Farrell's 2017 cheat sheet for Nielsen Norman Group divides a product cycle into four stages, and each stage has its own methods.
| Stage | Question | Typical methods |
|---|---|---|
| Discover | What do people need? | Field studies, diary studies, interviews |
| Explore | What shape should the solution take? | Competitor review, personas, prototype tests, card sorting |
| Test | Does the design work? | Usability testing, benchmarks, accessibility checks |
| Listen | What is happening now? | Surveys, analytics, search logs |
Her last stage, listen, continues after release, when teams watch for existing problems and for new ones. Listening is how a team notices that a shipped change did nothing, as the onboarding team's did. The cheat sheet also advises that when a budget cannot cover every stage, a team should do most of its research early, where findings change the most.
For the onboarding team, the discover stage would have meant watching a few new users start the app. The explore stage would have meant testing the three-screen idea on paper before building it. The team did only the last stage, and did it without an earlier finding to explain what it saw.
052 min
The common UX research methods
Within those stages a team has a short list of methods to draw on. Each answers a different question and has a known weakness.
| Method | What it answers | Watch for |
|---|---|---|
| ResearchUser interviews | ||
| ResearchUsability testing | ||
| ResearchSurveys | ||
| Not in the library yetContextual inquiry and field studies | ||
| Not in the library yetDiary studies | ||
| Not in the library yetCard sorting and tree testing | ||
| ResearchA/B testing | ||
| ResearchAnalytics | ||
| Not in the library yetExpert review |
Being watched can change behaviour. Researchers call this the Hawthorne effect, and it is the reason some teams prefer to observe people in their own setting for longer. A diary study asks participants to log what they do at set times, when a signal arrives, or when a chosen event happens. Eye tracking records where gaze lands, and a focus group is a moderated discussion with several people at once. Both are less common than the methods above and have limits that later sections explain.
The strongest studies pair methods. The onboarding team's analytics point to the screen where people stop. A round of interviews then asks why. A usability test of a redesign checks whether the fix works. No one method does all three jobs. Pairing methods also guards against the weakness of each one.
Every one of these methods raises the same practical question for the team: how many people do we need?
063 min
How many people a UX study needs
Where small studies came from
Teams once assumed that a serious study needed a lab and dozens of participants. Nielsen has said he built discount usability methods because he had a much smaller budget than well-funded labs. In 1989 he presented a paper on what he called discount usability. It combined simplified user testing with a small group, paper prototypes that support a single path through the interface, and heuristic evaluation, in which a person checks a design against usability guidelines.
How the five-user figure works
Small groups can work because the same problems keep appearing. In their 1993 model, Nielsen and Thomas Landauer found that one test user uncovers about 31 percent of a product’s usability problems, averaged over the projects they studied. Each person finds fewer new problems than the one before, since later users run into problems that earlier users already hit.
The arithmetic is short. One user misses about 69 percent of the problems. After five users, about 16 percent are still unseen, which is 0.69 multiplied by itself five times. After five participants, the model says a team has found about 85 percent of the problems.
Five per round, not five per study
The most common misreading is that five people are enough for a whole project. Nielsen's advice was the reverse. Nielsen advised spending a testing budget on three small studies of five users each, rather than one large study. The first round finds the biggest problems. The team fixes them, and the next round checks the fixes and finds what the first round hid.
Say the onboarding team follows this advice. They run round one with five new users, fix what they found, and run round two on the fixed version. The second round can now see problems that the first round's larger ones were covering.
When five is not enough
The rule has limits. Nielsen limited the rule to comparable users, so a product with several very different audiences needs people from each group. With two groups he suggested three or four people from each, and at least three per group so that the range of behaviour inside a group is covered.
Chance is the second limit. In Laura Faulkner’s 2003 study of 60 users, random groups of five found as many as 99 percent of the problems, and other groups of five found only 55 percent. A team that relies on one set of five could miss nearly half the problems. With 10 users, the weakest set in her study still found 80 percent, and with 20 users, 95 percent.
Quantitative studies are the third. A team that wants to report a number, such as the share of users who finish a task, is estimating a rate, and small groups give wide error. For a study that reports numbers, Nielsen recommended at least 20 users. Tighter ranges need more.
072 min
What people say and what they do
Sample size settles how many people to ask. It does not settle whether their answers are true. Say the onboarding team interviews a participant, who tells them they read the plan details every time they subscribe. The session recording then shows them tap straight past that screen. The participant is not lying.
Nielsen’s explanation is that when people describe what they do, they are really reporting what they remember doing. Memory leaves out the screens that felt unimportant, and a person's picture of themselves fills the gap. Wording, tone and the wish to look consistent push answers in the same direction. Researchers call that pull response bias.
The fix is a division of labour. Watch behaviour to learn what happens. Ask questions to learn why it happens, and treat the answer as a lead to check against what you saw.
Why focus groups do not answer usability questions
A focus group is a discussion with several people at once, so it looks like a fast way to hear many opinions. Nielsen wrote in 1997 that focus groups can only capture what customers say they do, not how they actually use the product. Participants usually watch a moderator demonstrate the product instead of using it themselves, and watching a demo is not the same as using one. In a group, people also hear each other and may hide confusion. Focus groups can still show what people want or how they feel about a brand. They cannot show whether a screen works.
082 min
How a UX research study runs
With the method and the number settled, the onboarding team can plan the study. The steps are the same for most methods.
- Write the research question and the decision it will change.
- Choose the method that answers that kind of question.
- Recruit people who match the real users, and screen out those who do not.
- Run the sessions. Ask open questions that do not lead, and give tasks that do not name the button to press.
- Analyse. For qualitative data, label each note with a code and group the codes into themes. For quantitative data, report measures such as task success, time on task and error rate. A short questionnaire such as the System Usability Scale turns satisfaction into a score.
- Report the findings ranked by importance, each with a recommendation, and say which findings are unwelcome rather than leaving them out.
- Act on the findings, then measure again.
Teams now use software to transcribe sessions and group notes into draft themes. That saves time, but a researcher still has to check each theme against the raw notes. A theme nobody can trace to a participant is a guess.
The ethics of studying people
Research on people carries duties. Participants should know what is being studied, agree to take part, and have their data protected. Companies can fail at this. Critics said the users in Facebook’s 2014 emotional contagion experiment had not been asked for informed consent and did not know they were taking part. Universities review studies through an ethics board, but private companies have no such general requirement. Larger teams therefore add a function called ResearchOps, which standardises consent forms, recruiting and templates so that good practice does not depend on one careful person.
091 min
A rule written into practice: the UK government service manual
Most teams treat research as a project. The UK government's service manual treats it as a starting rule. The UK government’s service manual tells teams to start by learning who will use a service, because without that they cannot build the right thing. It defines user needs as the needs a user has of a service, which a service must meet for the user to get the right outcome. It also says to treat any opinion or suggestion that does not come from users as an assumption that research has to prove.
That wording changes who has to justify what. In the onboarding team's planning meeting, the designer's remark that the screens were too long would have been logged as an assumption, with a study planned to test it. The manual also argues that well-researched services are more likely to be used and cost less to run, because less time and money go on fixing problems after launch.
101 min
UX research and the terms it is mistaken for
People use UX research as a name for several nearby things. Each one differs on one point.
| Term | What it is | The deciding difference |
|---|---|---|
| ResearchMarket research | ||
| ResearchUsability testing | ||
| Not in the library yetUser research | ||
| ResearchAnalytics |
The usability testing row is the common slip. Many teams say UX research and mean one round of usability tests. That covers the test stage of the cycle and leaves discovery and listening unwatched.
111 min
How UX research goes wrong
Research fails in specific ways, and each has a tell.
- The method follows habit. The team runs the survey it always runs. The tell is that every study ends with the same kind of finding, whatever the question.
- Surveys come before qualitative work. If a team has not yet talked to users, it does not know which questions to ask or which answer choices to offer.
- A solution is tested before the problem is confirmed. A usability test shows whether a design works. It cannot show whether the design solves a problem users have.
- Only the easiest users are recruited. If the team interviews only power users, or only friends of the company, the findings describe people who already cope with the product.
- Small groups are read as percentages. Three of five users failed is a sign of a problem, not a failure rate of 60 percent.
- The research confirms what the team wanted. Leading questions and friendly participants return the answer the team hoped for. The tell is a study that never surprises anyone.
- The findings sit in a deck. A report no one reads changes nothing. Short sessions that the team watches in person move decisions faster.
A final limit is scope. Research reduces uncertainty about users. It does not choose the product's strategy, and a study cannot replace a decision that someone must still make.
121 min
What to do with UX research on Monday
Start from the decision, not the method. Before asking for a study, a team should be able to finish the sentence If we learn X, we will do Y instead of Z. If no decision depends on the answer, skip the study.
The onboarding team's plan is short. Watch five new users start the app and note where each one stops. Interview them about what they expected the app to do. Test the fix on five more people before building it. Check the completion numbers after release, and keep the same loop running with small studies each sprint.
If the team has no budget, five existing customers and a paper sketch are enough to begin. If the team has a budget, it should spend most of it before the design is fixed. A team that does this swaps a stream of guesses for a stream of findings, and each release starts from what the last one taught.
?4 questions
Questions people ask
Do I need a dedicated researcher to do UX research?
How often should a team do UX research?
Can a team do UX research with no budget?
Which method should a team start with?
§10 sources
Sources
Jakob Nielsen, "Why You Only Need to Test with 5 Users", Nielsen Norman Group, 2000:
Jakob Nielsen, "How Many Test Users in a Usability Study?", Nielsen Norman Group, 2012:
Laura Faulkner, "Beyond the five-user assumption: benefits of increased sample sizes in usability testing", Behavior Research Methods, Instruments, & Computers, 2003:
Jakob Nielsen, "First Rule of Usability? Don't Listen to Users", Nielsen Norman Group, 2001:
Show all 10 sourcesShow fewer sources
Jakob Nielsen, "Focus Groups Are Not a Good Way to Evaluate Usability", Nielsen Norman Group, 1997:
Jakob Nielsen, "Discount Usability: 20 Years", Nielsen Norman Group, 2009:
Christian Rohrer, "When to Use Which User-Experience Research Methods", Nielsen Norman Group, 2022:
Susan Farrell, "UX Research Cheat Sheet", Nielsen Norman Group, 2017:
GOV.UK Service Manual, "Start by learning user needs":
"User research", an encyclopedia entry on methods, ResearchOps and ethics:
