Real Users vs. Synthetic: Why Human Insight Still Matters in Usability Testing

• Real users are still essential in usability testing and user research. AI-generated synthetic users cannot embody human behavior, emotion, or context.
• Synthetic users are useful, but in limited ways. Good use cases for synthetic personas are for desk research, pilot testing interview guides, early concept evaluation, and heuristic evaluation.
• The core risk of replacing synthetic users with real humans in usability is confusion between simulation and lived experience. Synthetic users are built from outsider data and patterns, not real insider perspective. With synthetic users, you risk receiving feedback that’s prone to people-pleasing.
• The best UX teams use AI to support research, not replace it. Let AI handle the predictable, repeatable work, and let real humans use friction to reveal potential innovation.

Eleven seconds.

In a usability interview, eleven seconds of dead silence feels like an eternity. I was concept testing a new onboarding intercept flow and watching a participant process the screen in front of her. We sat there together for those eleven, long, painful seconds of silence. The product team was observing in the next room and through streaming. No one moved. No one spoke.

Then she finally broke the tension with four words:

“I don’t get it.”

That silence showed us, in real time, where the design had failed to communicate. Before launch, the team had a clear signal exactly where the experience needed work.

These are the moments research with real users is built for.

In my thousands of hours conducting research, I have learned that the unpredictability of human behavior is where product innovation and 'aha' moments happen. It’s the long pauses, the shifting expressions, the moment when a user says one thing but their hands do another. It’s where a mix of observation and conscious probing surface the meatiest insights. 

And yet, there is growing pressure to replace all of this with something faster and tidier, like an AI persona standing in for a human being. Pressure to incorporate AI into every workflow is real, and user research is not immune to it. Organizations are increasingly turning to synthetic users to test concepts and validate designs driven by the promise of scale and efficiency.

What happens to those eleven seconds of silence when your user is an algorithm?

What is lost when we replace the messy, lived reality of humans with the sanitized instantaneous output of a predictive model?

No. AI cannot replace usability testing with real users. AI or synthetic users can support or accelerate some parts of the research process, but it cannot reproduce lived experience, embodied behavior, emotional friction, or genuine surprise from observing a user in context.

That distinction matters more now, not less.

AI tools can summarize patterns, generate likely responses, and mimic familiar user types. But usability testing is not just about collecting answers. 

Usability testing captures how a person encounters a product in context, and exposes the gap between intention (what a user says they want) and action (what a user actually does). Usability testing exposes the subtle, non-verbal cues that show confusion, hesitation, trust, discomfort, or relief.

Synthetic users are AI-generated profiles or simulated participants designed to mimic a user group. In practice, they are powered by large language models trained on large amounts of historical data about how people talk, behave, and describe their experiences.

It’s easy to understand the seductive appeal of synthetic users to a product team that’s moving fast, and being asked to do more, with less. They eliminate the friction and cost of finding, vetting, scheduling, and interviewing real human beings.

On the surface, research can appear cleaner, faster, and cheaper. However, convenience is not the same thing as validity.

To fully understand the limits of synthetic users, it helps to borrow a framework from anthropology: emic and etic.

The emic (insider) perspective

The emic perspective is the insider view. It reflects how people understand their own lives, needs, choices, and experiences. In UX research, this role belongs to the participant.

Users are the experts in their own lived reality.

The etic (outsider) perspective

The etic perspective is the outsider view. It reflects observation, interpretation, and analysis. In UX research, this role belongs to the researcher.

Researchers observe, probe, synthesize, and make meaning from what participants show and say.

Why the distinction between inside and outside perspective matters in user research

Good user research depends on both emic (insider) and etic (outsider) perspectives.

We need the participant’s account of what they feel, think, and intend. We also need the researcher’s observation of what actually happens. Together, these create a full picture of behavior.

Synthetic users disrupt that balance and can be wrong in ways that really matter to the outcome a study is trying to achieve.

Synthetic personas can act as 'people-pleasers,' validating concepts without demonstrating the critical friction that real participants bring if this tendency is not carefully addressed in their design, resulting in an over-reliance on LLMs. 

Synthetic users may:

• overstate clarity or confidence
• validate weak concepts too easily
• produce broad but shallow needs
• miss the role of context in decision-making
• fail to represent edge cases, contradictions, or emotional nuance

That makes synthetic users risky for concept testing or product validation, where false confidence can lead to expensive mistakes post-launch.

But the problem runs even deeper than agreeableness.

AI can capture the what—the pattern, the frequency, the aggregate of data sets—but it cannot reach the why, because the why lives inside a body, a history, a set of circumstances that no training data can fully encode. AI indexes on what is easily quantifiable across users, it can’t capture nuance, edge cases, or subtlety. 

It cannot feel the emotional weight of financial insecurity. It cannot hold two contradictory feelings at once and not know what to do with them. It cannot be surprised by itself.

So when a researcher asks a synthetic user how it feels, they are not getting real, human truth. They are getting an outsider summary dressed up as insider truth.

Human behavior is external, contextual, and messy

People respond to products in context. They adapt to distractions, habits, environment, stakes, memory, mood, and social pressure. They bring contradictions with them.

A participant may say a flow seems clear while still failing to complete it. They may claim a feature is useful but ignore it in practice. They may hesitate for reasons they cannot even explain.

Synthetic users are much better at generating tidy language than messy reality.

Real users reveal friction

In many usability sessions, the most useful insight shows up in a pause. A missed click. A moment of uncertainty only captured in the posture change of body language. These are the clearest signs that something in the design isn’t quite working.

Real users can surprise you

Strong product decisions rarely come from hearing what you expected to hear. They come from the participant who uses your product in a way you didn’t design for, or in an invented workaround the product team didn’t think of or plan for.

Surprise matters because that’s often where innovation begins.

Synthetic users do have some value to bring to the table in accelerating research, but they need to be used for the right jobs. 

It’s important to note that AI development and maturity is moving fast, but as of now, here are the best places for synthetic users in a modern workflow:

1. Desk research and early hypothesis building

Synthetic users can help researchers quickly synthesize what is already widely documented about a user group or domain. If you are entering a new space, they can help surface likely themes, language, and broad areas of inquiry.

This can be useful for:

• learning the basics of a domain
• identifying possible user motivations
• spotting common constraints or workflows
• drafting early hypotheses to test later with real users

The key is simple: treat the output as a starting point, not a finding.

2. Pilot testing interview guides

Before speaking with participants, researchers can use synthetic users to pressure test a discussion guide.

This can help you:

• spot vague or leading questions
• see where prompts may be too broad
• identify missing follow-up questions
• practice likely response paths

This saves time and helps sharpen the guide before real sessions begin. In this role, synthetic users are not replacing participants. They are helping researchers prepare to use participant time well.

3. Heuristic evaluation and baseline usability review

AI is very good at spotting known usability issues against established frameworks. It can flag violations of common heuristics, identify inconsistencies, and catch obvious friction before a design reaches users.

That makes it useful for:

• heuristic reviews
• baseline UX audits
• identifying low-hanging usability issues
• checking designs against predefined criteria

This is one of the strongest use cases because the task is already etic (outsider). It is about applying an external framework to a design, not claiming access to lived experience.

The best use of AI in user research is preparation, not replacement.

Use AI and synthetic users to optimize the predictable:

• summarize background knowledge
• sharpen study guides
• accelerate heuristic analysis
• organize early hypotheses

Then use real users to uncover points of potential innovation:

• friction
• confusion
• emotion
• contradiction
• adaptation
• surprise

That division of labor keeps research fast where it can be fast and human where it must be human.

As UX researchers, our role is not to reject new technology. Our job is to help drive innovation, after all. In this new era, we need to draw a careful boundary around how we use AI. 

Synthetic users can support the research process, but they can’t stand in for the people we design for.

If we replace human participants with simulations, we risk trading away the very thing that makes research valuable: the friction that tells us something we didn’t know before.

When the stakes are high for critical features, opt to sit with a real human being. Watch what happens. Listen past the first answer. Wait through the silence.

Sometimes the most important insight in the room arrives eleven seconds late.

Can AI replace usability testing?

No. AI can support research workflows, but it cannot replace usability testing with real users. Real usability testing reveals behavior, hesitation, confusion, and context that synthetic responses cannot fully reproduce.

What are synthetic users?

Synthetic users are AI-generated profiles or simulated participants that mimic a target user group. They are usually powered by large language models trained on historical data and are best used for limited research support tasks, not as substitutes for real participants.

When are synthetic users useful in UX research?

Synthetic users are most useful for desk research, early hypothesis generation, pilot testing interview guides, and heuristic evaluation. They are less useful for concept validation, usability testing, or any research that depends on real human behavior and lived experience.

Why are real users still important in user research?

Real users bring context, emotion, contradiction, and surprise. They show how people actually behave, not just how a model predicts they might behave. That is what helps teams make better product decisions.

What is the difference between emic and etic in UX research?

The emic perspective is the insider view of lived experience and belongs to the participant. The etic perspective is the outsider view of observation and interpretation and belongs to the researcher. Synthetic users are etic constructs, not true emic voices.

Should product teams use synthetic users at all?

Yes, but carefully. Synthetic users can be helpful when used as research support tools. They can catch low hanging fruit, like providing early iteration feedback to get a concept or prototype to a higher level of fidelity more quickly to prepare it to be tested with real users.

Synthetic users should complement real user research, not replace it.