By Karen Wang
What is Evaluative Research?
Evaluative research (also called evaluation research) measures how a product or concept performs when tested with real users, allowing us to road-test our hypotheses about how well things work.
Every time you try something new—an olive oil, a morning routine, a communication style—then compare your current results to your past ones, you’re doing a low-tech, layperson version of evaluative research.
From scientific research and psychology studies to healthcare systems, technology devices and consumer products, evaluative research can confirm how well a concept works, but it often reveals unseen roadblocks as well as opportunities for refinement and improvement.
What is UX-Specific Evaluative Research?
In terms of user experience, I’d define evaluation research as the scientific study of how something (a piece of technology, an application, or a concept) performs when groups of humans actually interact with it.
We use evaluative research to assess the success of a specific product or tool, to identify problem areas, and make evidence-based recommendations for improvements. I’d argue you can’t design truly breakthrough products, services, or experiences without it.
Evaluative research in a UX context focuses on specific technology products (prototypes, wireframes, or concepts). While each experience is different, we’re always seeking to see clearly and understand the user.
Generative vs. Evaluative Research: What’s the Difference?
Generative research is early-stage research that asks the big, broad questions. We use it to identify new opportunities and explore user needs (i.e., “What motivates people to sign up for a finance app?”).
Generative research is open-ended and foundational —it’s designed to uncover themes, rather than validate an existing concept. For example, when talking to participants about their shopping experiences, we might find that users’ core experience opportunities convene around browsing behaviors such as finding an item on sale.
Evaluative research is used later in the process after you’ve already zeroed in on a product or solution. You’re testing a particular thing—whether that’s a sketch, concept, wireframe, prototype, live experience, or an existing feature of a product, you have something in hand that you can put in front of users to see how they respond..
For example, an appliance brand client was releasing an update to its high-end smart refrigerator, so for one research phase, we observed participants stocking and organizing food in the new model just like they would do at home. (Spoiler alert: users cared more about icebox size than voice control.)
UX Evaluation Research: The Basics
Like any scientific research, UX researchers apply a sequence of steps to find answers to our questions. While every project is different, the sequence below is a quick overview of a typical evaluative research process:
- Define your goals. Like any scientific experiment, an evaluative research study requires clear goals or objectives. What are you testing? What is your hypothesis?
Sometimes we want to understand where friction arises in an experience—perhaps users are abandoning an e-commerce website. Sometimes we test a hypothesis (e.g., the purchase button is not discoverable) an independent variable (e.g., a wireframe or an onboarding process) or a dependent variable (e.g., the varying demographics and psychographics of different user groups).
If my goal is to understand where users experience friction during their check-out process, I might start defining my goals by assembling questions that zero in on potential friction points:
- What journey (steps) do users take when checking out their items?
- What information are users looking for as their journey progresses?
- What do users dislike or find frustrating about the experience?
2. Identify your users. Defining your audience accurately is crucial to getting accurate findings. We’ve done studies where our audience consisted of New England renters and homeowners trying to report a power outage, tech leaders who need to update their IT skills, or pregnant patients deciding whether to induce labor.
So when assembling your audience, you consider demographic information (e.g. age, sex, education level, income level, geographic location) but also the specific needs, aptitudes, attitudes and practices (a.k.a. psychographics) of your target users.
3. Choose your research methods. Focus groups, one-on-one interviews, diary studies…the methods toolkit is broad! While the aforementioned methods are fairly well-known (who among us hasn’t spent several hours in a conference room answering questions about yoga pants or movie trailers?) here are some other tools we use:
- Card sorting is a categorization exercise where participants sort topics into categories, which we use to discover patterns in how participants classify information.
Let’s say an online retailer wants to organize its product categories better. We could have participants sort cards labeled with product names (e.g., “running shoes,” “yoga mats,” “water bottles”) into groups that make sense to them, so we can develop product categories and filters that match users’ mental models and make it easier to find what they need.
- Benchmarking is the process of evaluating and comparing products or systems using metrics to gauge performance.
There are different categories of benchmarking, as well: with competitive benchmarking, for example, we measure user responses to products against similar ones on the market. A smartphone manufacturer might ask us to test focus groups’ reaction to their new device’s battery life, camera quality, and durability against competitor models. We also do internal benchmarking that evaluates the same product using established UX metrics over time.
- Usability testing allows us to observe users as they complete real-life tasks using a product or service to discover pain points and opportunities for improvement. Usability testing lets us engage participants directly to understand their subjective experiences and to observe their unspoken behaviors. We might watch new users as they complete the sign-up and tutorial process for a fitness tracking app, with the goal of streamlining onboarding flow to reduce drop-offs and improve retention.
- A/B testing compares audience responses and outcomes to two different choices: A or B. (Anyone who's ever had their vision checked has been the subject of A/B testing.) We might test which version of a website purchase button color and call-to-action (CTA) copy gets more clicks and conversions: a green button that says “Buy Now” or a red button that says “Get Yours Today”?
- Mixed methods research combines several different modalities to give us a deeper, fuller picture of the user experience, reducing potential biases and strengthening the validity of your results.
4. Recruit your research participants. Finding participants who are representative of the audience that will actually use the product, service, or concept you’re evaluating is key. Experienced recruitment firms typically have an extensive database of potential candidates that they can sort based on demographics, behaviors, and professions that match your target audience—from new parents, IT specialists and medical patients to gaming enthusiasts and leisure travelers. Good researchers know it’s also essential to build rapport with your participants, as they share their most insightful feedback when they feel comfortable, safe, and respected.
5. Conduct your research. Here’s where we get to observe how participants interact with the idea, product or service we’re testing. While some research can be done remotely (and some projects really benefit from remote research) others require specific settings and tech capabilities in order to get the most accurate results.
In-depth research often requires audio recordings, transcriptions, broadcast-grade streaming, note-takers, and on-site IT support, not to mention experienced moderators and facilitators. Most companies don't have these capabilities or staffing, much less labs that can be configured to simulate an office, living room, bedroom, hospital room, or kitchen—which is why our Seattle usability labs provide all of this technology and personnel, plus comfortable observation rooms for client teams to observe sessions.
Your research setting matters, and in many instances fully staffed state-of-the-art research facilities do provide better data collection.
6. Analyze your data. By interpreting the information you gathered, you can understand the story your data is telling.
7. Present your findings. Sharing your findings with team members, clients, or decision-makers typically highlights key takeaways, recommendations, and next steps. What’s the next best action to make use of your findings?
Why Evaluative Research Requires Experienced Practitioners
Evaluative research is only as good as the practitioners doing it. Even deciding whether you need A/B testing, focus groups or one-on-one interviews requires experience and specialized knowledge. Getting the clearest, most accurate information from your audience participants is both an art and a science, so formal training is key.
Unconscious biases, echo chambers and leading questions can detract from the accuracy of the responses we get, so we take extensive precautions to neutralize our queries, listen to, and empathize with the user. Because we need accurate answers to make informed decisions, seasoned practitioners know how to ask open-ended questions and guard against bias. A choice as simple as flipping A/B to B/A can change participant responses.
Ultimately, good research empowers people and teams to make informed, evidence-based choices that lead to better outcomes. At Blink, our evaluation process for UX always stipulates that we are not testing users, we are testing the product, prototype, or service. We use scientific methodologies to remain unbiased and we always create a holistic experience end-to-end—which means we engage with stakeholders early and often to inform our methodologies, objectives, and findings. We take great pains to incorporate every facet of a product team and beyond to ensure we're not stuck in an echo chamber.
One of the best parts of my job is that everyone around me is truly an expert in their field, and our combination of synergy, creativity, and autonomy results in work of the highest caliber. Our ability to go broad in order to converge on high-impact recommendations really sets us apart. I love working in a setting where the collaborative depth of our teams means we deliver evaluative research that’s not only rigorous, but deeply resonant and actionable. Our results are always more than just the sum of our parts.
Karen Wang is a UX Researcher at Blink who specializes in empathetic perspective-taking. Her passion lies within studying the nuances of exceptional user-centered design, particularly through co-creation methodologies. She received her Master’s in Human Factors in Information Design from Bentley University in Boston.
Is the solution you’re building on the right track? Our expert teams and our state-of-the-art research facilities have the answers you’re looking for. Let’s talk!
