From a business perspective, pursuing data-driven decisions has never been more critical. Companies collect vast amounts of data and track numerous metrics, yet they often believe large sets of data are necessary to yield valid results, which isn’t always the case.
By understanding the differences between qualitative and quantitative data and assessing your user or market research goals, you can calculate how many participants you need for a research study — and it may be fewer than you think!
What is quantitative versus qualitative data?
Most people are familiar with quantitative data because it is numerical. It helps us answer questions about what users are doing or thinking, as well as questions about scale (how much) and frequency (how many). To get reliable and generalizable answers with quantitative research, you typically need larger sample sizes, aka more participants.
For example, you work at a bank and want to understand customers’ satisfaction with your mobile app. You survey app users to find out how much satisfaction they have with the app and how many people are satisfied.
After viewing the results, you find that 25% have low satisfaction (the “what”). But why are they unsatisfied? And more importantly, what can you do to improve the experience? That’s where qualitative data comes in.
Qualitative data, which is descriptive in nature, helps us understand the why behind user behaviors and feedback. For example, by observing customers using your mobile app, you can uncover where they struggle and start to identify potential solutions. If you’re interested in understanding why customers don’t like your app (versus how many don’t like it), qualitative research can get you reliable results with a smaller sample size or fewer participants.
Sound research doesn’t always require statistical significance.
Nothing kills a researcher’s spirit faster than these two questions: “Is it statistically significant?” and “How many participants did you talk to?” In reality, aligning sample sizes with your research goals is more important than statistical significance or the absolute number of participants. There are instances where it’s necessary to gain statistical significance — such as experimental designs — but it isn’t always needed. Let’s look at some of the common goals and methods of user research and how this can help you determine your sample size.
Foundational Research
Sample size: 10-12 participants
Methods: User interviews, diary studies, contextual/field research
Goal: Gain a deeper understanding of your topic or area of interest
Foundational research usually occurs early in the design process. It helps you define the problem space and user needs to guide strategy. For foundational research to be successful, you’ll want to hit saturation (a fancy research term that means you’ve learned most of what you can about your topic, so talking to more participants won’t give you any new information). If you start hearing participants say the same things, it’s a good indicator you’ve hit saturation.
Factors like multiple user groups or geography can affect the number of people you need to hit saturation. For example, if you’re researching IT managers and knowledge workers in a specific industry, you might need to interview 10 people from each group. At Blink, we aim to find the smallest sample size, typically 10-12 participants, required to hit saturation. Interviewing more people than necessary can increase research costs without providing much extra benefit.
Formative UX Research
Sample size: 6-8 participants
Methods: Concept/prototype testing, RITE, participatory design
Goal: Identify issues and considerations early on to inform the design before starting development
Formative evaluative research happens later in the product life cycle and assesses how well new concepts and designs work for users. It costs 100x more to fix a bug after launch than before, so uncovering issues early can significantly reduce post-launch costs.
Similar to hitting saturation, there comes a point when testing with more users doesn’t discover new issues. We usually reach this point for formative studies after testing with 6-8 participants. Nielsen Norman Group provides an excellent visual to show the diminishing returns based on their formula for uncovering usability issues.
Summative UX Research
Sample size: 15+ participants
Methods: Baseline and benchmark usability testing
Goal: Measure the user experience
Baseline and benchmark usability testing are both types of summative research that help evaluate a product’s performance and user experience. Baseline testing typically happens at the beginning of a project. Here’s what it could look like using our mobile banking app example:
Let’s say you learn that people have trouble depositing mobile checks and checking their account balance on your app. To fix this, you decide to redesign those features. First, you conduct baseline research to test the current app’s usability as a starting point, measuring metrics like task completion rate (how many people can complete these tasks) and time on task (how long it takes). After the redesign, you would repeat the same research study to see if the design changes improved your metrics.
On the other hand, benchmark testing is an ongoing process that involves measuring user experience over time. For example, you might compare your mobile banking app’s performance with other similar apps to assess how well you perform compared to competitors or to track whether your UX is improving or declining over time.
Determining the right metrics and stats parameters for your summative research (e.g., margin of error, confidence intervals) is an important part of the process and necessary in calculating the sample size. Common UX measurements include task completion rates, time on task, and the system usability scale (SUS).
We recommend a minimum of 15 participants for these types of studies; however, you may need more if you desire a high level of precision with low risk (high confidence interval and low margin of error). Jeff Sauro and James R. Lewis wrote a book on quantifying the user experience, a great resource for further digging into the topic.
Survey Research
Sample size: 100+ participants
Methods: Online surveys
Goal: Understand your target population’s attitudes, sentiments, and self-reported behavior and/or compare attributes
Depending on your research objectives, survey research can occur at any point in the design process. One key reason to run a survey is to be able to generalize what you find from your sample (people who respond to the survey) to a broader audience (the population size or total number of users).
Due to this, statistical significance is important. As a general rule of thumb, aim for at least 100 participants, but you may need more than that to have a statistically significant sample size.
Estimate your sample size by using your overall target population size (if you know it) or conducting a power analysis (to find it). Qualtrics’ sample size calculator is a great tool to help you with your specific project.
Right-size your sample
So, how many people do you need to talk to for valuable and trustworthy research results? It depends on your goal. Start by figuring out what you want to achieve with your research, then choose the appropriate research method; this will give you a good idea of your sample size.
When it comes to qualitative user research, you can often achieve reliable, cost-effective results by conducting fewer than 12 high-quality interviews. Use this guide as a start, and reach out to Blink UX if you have questions or want to discuss research for your next project.