Focus on What Users Do, Not What They Say
Understanding your users can help you better serve them. One way to learn more about how your users respond to your design is to ask them their opinions during a usability test. While this can be a fast and easy way to get user data in a session, there are five reasons to be skeptical of verbal feedback during usability testing:
1. Participants don’t want to hurt your feelings. Participants in usability studies often want to be nice to you and may be afraid they will hurt your feelings or get someone in trouble if they criticize the interface being tested. To this point, I’ve noticed that some participants more readily tell me about things they like in an interface than things they don’t like.
2. Participants want to appear more skilled than they are. I’ve seen study participants struggle on tasks and then tell me it wasn’t that hard when I’ve asked them about their experiences. These responses are understandable; participants are being observed in a usability session and they may want their audience to see them as competent, technically savvy individuals.
3. People are generally bad at predicting their future behavior. When asked about their future behavior, study participants do not know what their future context will be and how they will act within it. It is this context, rather than their current opinions, that will most influence their future behavior.
4. Participant desires can lead to feature overload. If you added every feature that study participants said they wanted, you could end up with a cluttered interface that diluted the most valuable part of your product and made key features hard to find.
5. Users are not designers. Users can help you identify problems with your interface within the context of a usability test, but unless they are trained designers, their input alone should not be relied upon to redesign it.
The most actionable findings from a testing session usually come from what study participants do.
As Jakob Nielsen mentioned many years ago, “pay attention to what users do, not what they say.” Unlike opinions, behaviors are honest. If a study participant cannot complete something on a website, this directly reveals a problem area. In contrast, if a participant dislikes something on the site, there may be no problem at all, because other users may love it.
Be courageous and take a stance with your design. Not everyone will like everything about your design, and that is okay.
When you gather opinions, uncover the why.
Meaningful insights can be gathered from user opinions, but this often requires additional follow-up questions to understand why users feel a certain way. Understanding the why can reveal underlying needs that the interface is or is not meeting, and can help you learn how to improve the design.
As an example, I was testing a website on a desktop computer and a user said he wanted horizontal scrolling on the home page. In many cases horizontal scrolling is a poor design choice and would have been so here. After talking to him further, I learned that his underlying pain point was that it took too long to find information – not that he couldn’t horizontally scroll. The site required a lot of scrolling to see all of the information on the page, and it wasn’t clear to him where to go on the site to accomplish different tasks. Understanding this underlying pain point helped me provide a design recommendation that improved the user experience—and you can bet the recommendation wasn’t to add horizontal scrolling.
User feedback does have a place in usability testing, but it’s most informative when backed up by user behavior.
Brian Essex works in user research at Blink UX, joining the team after attaining a Ph.D. in Cognition and Cognitive Neuroscience from Vanderbilt University.