# Heuristic evaluation report: severity and priority for each problem

> A heuristic evaluation report documents each problem with its location, heuristic, severity, and fix. Learn the 0-to-4 scale and how to write useful findings.

*Tags: ux, artifact, research, usability, prioritization*

---


> [!info] Quick Definition
> A **heuristic evaluation report** documents usability problems found when reviewing an interface against recognized principles, and assigns each one a **severity** that indicates how urgently it should be fixed. Its value lies in letting anyone on the team reproduce the problem and understand why it matters.


## What is the report?

It is the deliverable of a [heuristic evaluation](/en/wiki/techniques/heuristic-evaluations/). A group of evaluators walks through the interface looking for violations of heuristics, such as Nielsen's ten (first published in 1994 and essentially unchanged), and logs each finding. The report turns that loose list into something the team can prioritize and fix.

A common mistake is confusing the report with a list of opinions. A good report describes observable behavior: what happens, where, with what consequence, and under which principle.

## How is severity rated?

The most widely used scale is Nielsen's 0-to-4 scale:

| Value | Label | What it means |
|---|---|---|
| 0 | Not a problem | Noted, but does not affect the experience |
| 1 | Cosmetic | Fix if there is time |
| 2 | Minor | Low priority |
| 3 | Major | Important to fix, high priority |
| 4 | Catastrophe | Must be fixed before launch |

Nielsen explains that severity combines frequency (how many users run into it), impact (how hard it is to overcome), and persistence (whether it keeps bothering users after they know about it). He adds market impact: a problem that is easy to overcome can still hurt a product's popularity. MeasuringU uses its own three-level scale, minor, moderate, and critical, based on impact on the user, and warns that the numbers attached to each level are somewhat arbitrary: what matters is the order.

That warning applies to any scale. Do not look for decimal precision where there is only judgment.

## How do you combine ratings?

One person is not enough: Nielsen Norman Group considers a single evaluator's ratings too unreliable to trust. MeasuringU recommends having several evaluators rate severity independently, then calculating agreement and averaging. Nielsen notes that the mean of three evaluators is satisfactory for many practical purposes.

A procedure that limits bias:

1. **Each evaluator rates in silence**, before any group discussion.
2. **Ratings are compared**, and only large gaps, for example two points or more, are discussed.
3. **The reasoning behind the highest ratings** is recorded, especially the 4s.
4. **The average is kept.** If two evaluators report the same problem separately, CleverX recommends merging them into one finding and keeping the highest severity assigned.

Nielsen warns that it is difficult to get good severity estimates during the session, because evaluators are busy looking for new problems. That is why ratings are usually collected at the end with a questionnaire, from the full list of problems with their descriptions.

## What should each finding include?

A useful entry has at least these fields:

- **An observable description**, in one or two sentences: what happens, not what the evaluator thinks of it.
- **Location**: the screen, flow step, or component.
- **Heuristic violated**: for example, "Help users recognize, diagnose, and recover from errors".
- **Severity** on the 0-to-4 scale.
- **Evidence**: an annotated screenshot, a clip, or a timestamp from a session.
- **Recommendation**: a concrete, actionable fix.
- **Evaluator**, so you know who found it and can ask them.

Koji suggests attaching evidence to every rating above 2. That rule keeps serious problems from being debated on arguments alone.

## How should the report be organized?

A table with one row per problem is the simplest format. Nielsen Norman Group's guide suggests capturing one observation per line with its heuristic and, once the independent evaluations are done, clustering similar problems with an affinity diagram before discussing which are most harmful and which need more data.

For the final report, it helps to sort findings by severity, most serious first, rather than by heuristic number. That way the team starts with what has the most impact.

The executive summary should fit on one page: the study goal, the method, the three to five most critical findings, and one main recommendation. Anyone reading only that summary should know what to do.

## Is severity the same as priority?

No. Severity describes how much a problem hurts the user. MeasuringU recommends treating frequency separately from severity: a serious problem almost nobody hits and a mild one everybody hits are not ordered the same way. Nielsen's scale already hints at priority in its labels (level 2 is low priority, level 3 is high priority, and level 4 must be fixed before launch), but the final call on what to fix first adds variables the report does not measure, such as the cost of the fix. So the report should deliver severity and evidence, and the product team should decide the order.

## Example

Illustrative example: in an evaluation of a ticket-buying app, four evaluators find that when a payment fails, the message only says "Error 500". The first rates it 3, the second 4, the third 3, and the fourth 4. When they discuss the gap, they discover that the second evaluator tested with a valid card that the bank declined, and the message did not explain what to do. The finding reads: "When a payment is declined, the message does not say whether the problem is the card or the system; it violates the heuristic about helping users recover from errors. Severity 4. Recommendation: show the reason for the decline and the two possible next actions." The evidence is a 20-second clip of the failed attempt.

## Common mistakes

- **Describing opinions** instead of observable behavior.
- **Discussing severity as a group before** each evaluator has rated in silence.
- **Leaving the recommendation vague**, such as "improve the experience".
- **Reporting problems without evidence**, which makes them impossible to reproduce.
- **Using severity as a substitute for testing with people**: heuristic evaluation does not replace usability tests with real users.
- **Delivering a long list with no summary**, which nobody reads.

To test with people the problems the evaluation detects, [usability testing](/en/wiki/research/usability-testing/) is the natural next step. If the report comes from comparing products, the [competitive matrix](/en/wiki/artifacts/competitive-matrix/) helps place the problems against competitors. And to decide what to fix first among several priorities, the [[kano-model|Kano model]] separates what is basic from what differentiates.

## Sources

- [Usability Issue Severity Ratings: How to Score, Prioritize, and Report UX Problems, Koji](https://www.koji.so/docs/usability-issue-severity-ratings)
- [Rating the Severity of Usability Problems, MeasuringU](https://measuringu.com/rating-severity/)
- [Heuristic evaluation template: a ready-to-use framework for UX reviews, CleverX](https://cleverx.com/blog/heuristic-evaluation-template/)


---

Source: https://www.fernandoux.com/en/wiki/artifacts/heuristic-evaluation-report/
