Heuristic Evaluation

011 min

Heuristic Evaluation at a glance

• What it is: Evaluators independently check an interface against 10 usability principles, then pool results.
• Origin: Nielsen and Molich introduced the method in 1990; Nielsen refined the 10 heuristics in 1994.
• Evaluators needed: Three to five — a figure Nielsen derived from cost-benefit modelling of his own past projects.
• Backfires when: run alone. Experts and real users flag different problems, so it misses roughly half of what user testing finds.

021 min

How Heuristic Evaluation works

Heuristic evaluation asks a small group of evaluators to inspect an interface on their own, before anyone compares notes. Each evaluator goes through the interface at least twice: once to get a general sense of the flow, then a second pass checking individual screens and controls against ten usability principles — Nielsen's heuristics, covering things like whether the system shows its current status and whether a mistake can be undone.

When an evaluator finds something that breaks one of the ten principles, they write it down, name which principle it breaks, and give it a severity rating: how much it would slow someone down or stop them completing a task. This step happens alone and in writing, not in a group discussion, on purpose. Evaluating together first would let one evaluator's opinion set what everyone else looks for, a version of confirmation bias the method is built to avoid.

Only after every evaluator has finished independently does the group meet to combine their lists. No individual evaluator, no matter how experienced, catches every usability issue alone. Different evaluators tend to catch different problems, so the combined list from several evaluators includes more real problems than any one evaluator's list by itself.

031 min

Where Heuristic Evaluation comes from

Jakob Nielsen developed heuristic evaluation together with Rolf Molich in 1990, while both worked as usability consultants. Their original heuristics grew out of guidelines they had already been using in consulting work, checked against interfaces they had actually evaluated rather than written from theory alone.

Four years later, Nielsen rebuilt the list from a different kind of evidence. He collected 249 usability problems from 11 earlier projects, then measured how well each candidate heuristic explained the problems that had actually been found. That analysis produced nine heuristics. A tenth, Help and Documentation, was not part of that statistical result — Nielsen added it back afterward, and the ten heuristics have stayed the same since 1994.

042 min

The 10 heuristics behind Heuristic Evaluation

Nielsen's ten heuristics are broad rules an evaluator checks an interface against, not specific guidelines written for one particular screen. Each one names a property a good interface has, stated as something an evaluator can check for directly.

The same ten checks apply to almost any interface. None of them names a specific screen or technology, so they work as well for a mobile app as for a spreadsheet or a physical kiosk.

052 min

How many evaluators a Heuristic Evaluation needs

Nielsen recommends three to five evaluators. That number comes from his own cost-benefit modelling of past projects, not from an outside study measuring the same question on different projects.

The case for more than one evaluator starts with how little a single evaluator finds alone. Averaged across six of Nielsen's own projects, single evaluators found only 35% of the usability problems in the interfaces they reviewed. Individual find rates in those six projects ranged from 19% to 51%, averaging 34%, on interfaces that held 16 to 50 total problems, averaging 33. Aggregating several independent evaluators fixes this, because different evaluators tend to catch different problems.

The case for stopping around five, rather than adding more, is about cost. Based on figures from his own projects, Nielsen estimated a heuristic evaluation's fixed cost at $3,700 to $4,800, plus $410 to $900 per additional evaluator. Applying those numbers to one sample project, a fourth evaluator brought the total cost to $6,400 and found usability problems Nielsen valued at $395,000. Past that point, each additional evaluator kept costing the same amount while finding progressively less that the others had not already found.

Both halves of this argument are Nielsen's own estimates from his own consulting work. Nobody has independently repeated the cost-benefit study on a different set of projects, so three to five is best read as Nielsen's own calculation of where the return stays high, not a number an outside study reproduced.

061 min

How Heuristic Evaluation shows up in practice

The clearest evidence for what a heuristic evaluation actually turns up comes from Nielsen's own record of running the method. Across six real interfaces in Nielsen's own case studies, heuristic evaluation identified a total of 59 major usability problems and 152 minor usability problems. Most of what shows up is minor: a label that reads unclear out of context, an icon two people would read two different ways, a confirmation step where none was needed. Problems serious enough to stop someone completing a task showed up too, just in smaller numbers, which is why every problem found gets a severity rating instead of being treated as equally urgent.

In a product team, a heuristic evaluation usually runs before a design ships and before a usability test is scheduled, not instead of one. A designer and an engineer check a new feature against the ten heuristics before it goes live, fix what an expert can catch on sight, and still set aside time to watch someone unfamiliar with the product try to use it.

071 min

How to run a Heuristic Evaluation

1. 1

   Define the scope.

   Decide which flows or screens the evaluation covers. A full evaluation of every screen in a large product takes too long to stay useful.

2. 2

   Brief the evaluators.

   Walk them through the ten heuristics and the interface's intended users and main tasks, without telling them what to look for.

3. 3

   Send evaluators through alone.

   Each person goes through the interface at least twice: once for a general feel, once checking specific elements against each heuristic.

4. 4

   Record problems as they're found.

   Tag each one with the heuristic it breaks and a severity rating, and don't discuss it with the other evaluators yet.

5. 5

   Combine the lists.

   Once everyone has finished independently, merge the lists and talk through any disagreement about whether something is really a problem.

6. 6

   Prioritize by severity

   , not by how many evaluators happened to notice the same thing. A problem only one evaluator caught can still be the most serious one on the list.

082 min

Where Heuristic Evaluation falls short

Heuristic evaluation and usability testing do not reach the same problems by two different routes. They find different problems. An evaluator inspects a design against a checklist, using their own mental models of how an interface should behave. A user in a usability test is not inspecting anything. They are trying to finish a task, and a problem only shows up if it actually stops them partway through. An expert looking at a screen and a person trying to get something done are doing two different things, so heuristic evaluation and usability testing end up surfacing two different sets of failures.

One direction of that gap is false positives. Sauro's 2012 review of five published comparisons found that on average, 34% of the problems evaluators flagged in a heuristic evaluation were never seen when real users were watched doing the same tasks. Evaluators called these problems. The people actually using the product never ran into them.

The other direction is misses. The same review found heuristic evaluation turns up only around 36% of the problems a usability test finds. It misses roughly 49% of what watching real users uncovers. None of the evaluators were the person actually trying to complete the task, so those problems never came up during the evaluation.

Neither number makes one method better than the other on its own. It means a design reviewed only by experts still carries every failure that only shows up once someone tries to use it for real.

092 min

Heuristic Evaluation vs. usability testing

Both methods exist to catch problems before a design ships. What separates them is who does the judging and how.

The two methods do different jobs, not the same job twice. A heuristic evaluation is cheap enough to run early and often, catching the kind of problem an expert can spot on sight: a missing status indicator, an inconsistent label, a confirmation step with no way to undo it. A usability test is the only way to see whether people can actually use the thing, because it is the only one of the two where an actual user takes part.

102 min

Frequently asked questions about Heuristic Evaluation

?7 questions

Questions people ask

§9 sources

Sources