Ask the experts: Why heuristic evaluations are vital to your user testing

By RocketAir Crew
June 15, 2020

Everywhere you look, the odds always seem stacked against you. Whenever there’s a Powerball jackpot that starts to reach epic levels, for example, you’ll inevitably read the stories telling you that your chances of winning are beyond absurd. Or maybe you’re a college athlete who dreams of making it to the pros, landing that shoe deal, having more money than you’ll ever be able to spend in one lifetime? Yeah, good luck with that. And, of course, you’ve quit your job and launched that startup you hope will be seen as so cutting edge that VCs will be tripping over themselves to throw bags of cash at you. Except, odds are more likely that you’ll be one of the 90 percent of startups that fail.

There can be multiple factors as to why you don’t succeed in any given situation: It could be just those ridiculous odds—like, say, a 1 in 292,201,338 chance of winning a Powerball jackpot—that makes failure the only reasonable outcome. If you’re a startup, it could be things you can control—such as management, staffing, product development, cash flow—or things that are out of your hands, like watching the economy tank just as you launch your company.

But for the sake of this blog, we’re going to focus only on the things that you and your team can control, specifically as it relates to product design. Sticking with the theme of odds, did you know that 24 percent of apps are only ever used once? This accounts for roughly one-fourth of all apps that are brought to market.

This means that a significant number of apps have failed at user retention, often because of failures in design. And so, in order to increase your odds of success, it’s important that you and your team conduct a heuristic evaluation in the early stages of your design and development (although they can be conducted multiple times at any stage of the design process) in order to better ensure that your users will actually find your product user friendly.

10 rules of thumb

Heuristics are essentially considered rules of thumb, or a set of criteria by which to evaluate your website or app design. What makes a heuristic evaluation different than usability testing is that rather than sourcing out users to test your product like with the latter method, the former relies on (roughly) 1-3 experts to determine how your design complies with a set of heuristics to get a better picture of its usability.

As for the heuristics that should be tested, the industry standard was set by Jakob Nielsen of the Nielsen Norman Group, who established the 10 Heuristics for evaluating the usability of your UX and UI design. They are as follows:

  1. Visibility of system status. Users should always be informed of what the system, or product is doing, in a reasonable timeline.
  2. Match between system and the real world. Avoid technical jargon. The product interface should align to terminology and language familiar to the user.
  3. User control and freedom. Mistakes can happen, provide a way out when they do. Supporting undo and redo is one way of achieving this, along with methods of cancelling an operation.
  4. Consistency and standards. Don’t reinvent the wheel. Users understand conventions and patterns and should not have to guess as to what something means. Follow patterns that exist across platforms.
  5. Error prevention. Providing feedback when an error occurs is important, but better yet if you can help prevent an error from occurring in the first place. Providing smart defaults, and confirmation is beneficial.
  6. Recognition rather than recall. To maximize usability, reduce the load on memory and recall. Avoid hiding important actions behind menus, and instead present options in a visible way so users don’t have to remember where they are.
  7. Flexibility and efficiency of use. Accommodate both novice and advanced users, but tailor the experience. Provide ways of speeding up workflows, and accelerating users familiar with the system, while guiding those less familiar.
  8. Aesthetic and minimalist design. Don’t overwhelm users by displaying unnecessary information. Keep screens and dialogues focused and minimal to maximize visibility and clarity.
  9. Help users recognize, diagnose, and recover from errors. Errors will occur, and it is important to help users understand what caused the error, in plain language, and how they can go about rectifying the situation.
  10. Help and documentation. Though it is best to design away the need for help and documentation, it is important to make it accessible when it is needed. Don’t make users struggle to find help, and where possible present it contextually as needed, in plain, clear language.

Although Nielsen’s 10 Heuristics have become the standard bearer for evaluating usability—and are very good rules of thumb to follow—Matt Rea, a product designer at Zenreac, points out that that Nielsen established these in the ‘90s (they appeared in his 1994 book Usability Engineering), and so your evaluation may require additional heuristics that are more specific to your product design.

“These heuristics provide a strong core however they were written in the early 90s, at a time when technology products, especially software, were in a very different state,” Rea explains. “Though for the most part, these hold true today, it may be beneficial to review these and reflect on whether they’re the right heuristics to use for your own particular product.”

It’s generally thought that a heuristic evaluation is best conducted by more than one expert (at least three is Nielsen’s recommendation) rather than relying on a single person to test your product’s usability.

“It is usually conducted by a group of experts because it is very likely that one person will not be able to find all usability problems,” product designer Fabio Muniz wrote for UsabilityGeek. “On the other hand, a group of different people tend to analyze an interface from different angles and as a result are more likely to identify a wider set of areas for improvement.”

And the reason that you bring in some outside experts to conduct the heuristic evaluation is because they “are not connected in any way with the company so their feedback is impartial and free of internal politics,” as content marketer Lionel Valdellon notes.

“This doesn’t tie up any internal resources,” Valdellon adds. “Once the experts provide their feedback, the development team can implement immediately.”

Another important point: The team of experts should conduct evaluations individually so they can arrive at their own conclusions independent of the other experts. You should also have someone who is familiar with the product that can be on hand to document the feedback and answer any questions.

UX designer Dawn Schlecht explains:


The benefit of having multiple reviewers is that although they will likely catch many of the same errors, they each will likely find some others have missed. And the number three is important because, although with more people reviewing the site you may find more errors, it's not likely to be statistically significant.


Good news, bad news

Okay, so first the good news: Not only is a heuristic evaluation an important step in catching obvious usability issues early in the development process, but it is normally a lot less time-consuming and expensive than user testing. Now the bad news: A heuristic evaluation on its own is really not enough to test the usability of your product, which means that—as Nielsen recommends—it should be conducted in conjunction with user testing.

Because a heuristic evaluation is normally done early in the design process—sometimes even when the product is still on paper—it’s important to have users test the product when it's closer to completion. Another potential drawback to the heuristic evaluation is that your testing criteria are rules of thumb deemed important to you (even if you’re using Nielsen’s 10 Heuristics), but users could present you with usability issues that you hadn’t considered.

“Both will largely uncover different insights and errors to be corrected,” Schlecht writes. “Ideally, you would want to do both at several different stages of development.”

Also, while some experts cite costs as an advantage that heuristic evaluations have over user testing, this should not be confused with thinking of heuristic evaluations as cheap. They can, in fact, get expensive especially when you’re paying to bring in multiple experts.

So, if you’re a small startup design firm or a freelance designer, how can you evaluate the usability of a product if you can’t afford to bring in experts? Schlecht points to a method called Heuristic Markup that’s presented by Leah Buley in her book The User Experience Team of One.

In a Heuristic Markup, you are the expert, meaning that “you set aside several hours to walk through the product yourself,” as Schlect notes. She explains:


Take yourself through the product, from beginning to end, as you think a user might do. You might try using one of your personas and/or thinking of the journey your users may take through the site as they try to accomplish specific tasks.


Buley then “suggests taking a screenshot of each step of the journey, pasting it into a PowerPoint and then making notes of your observations, while you are in your user’s mindset,” and making note of your reactions while you go through the design.

Final thoughts

A heuristic evaluation can be conducted at any stage of the UX design process, but it’s best to conduct it early in order to catch any glaring or obvious usability issues that would present themselves with the completed product. It’s also important to bring in multiple experts in order to get a more well-rounded assessment of your design and determine what works and what doesn’t.

But you must be cognizant of the limitations inherent in a heuristic evaluation: While it will provide valuable, expert insights into your product, you cannot get a complete picture of a product’s usability until you test it with the people who will actually be using it.