Synthetic Users Are Not Real Research: What AI Can and Cannot Do in UX Testing

A startup called Synthetic Users launched with a simple promise: stop paying $100 per participant and waiting weeks to schedule sessions. Just describe your target user, and the AI will simulate what they would say, what they would struggle with, and what they would think about your product.

The design community had two reactions. Half were excited. Half were furious.

Both reactions were understandable, and both missed the point.

Synthetic users are AI systems trained on internet-scale behavioral data, interview transcripts, and user psychology research. They can simulate responses from a described persona: "a 45-year-old nurse in a rural area who uses a smartphone but is not technically confident." You describe your product or show it a prototype, and it generates responses that sound like they came from that person.

The tools doing this now include Synthetic Users, Persona Labs, and a growing number of research platforms adding AI simulation as a feature. Even established tools like Maze and UserTesting are experimenting with AI-generated supplementary feedback.

The pitch is compelling, especially when your research budget is zero and your timeline is this week.

Here's what's real and what isn't.

What Synthetic Users Are Built On

To understand the limits, you need to understand the mechanism.

Synthetic user systems are large language models, the same underlying technology as ChatGPT and Claude, fine-tuned on user research transcripts, behavioral science literature, consumer psychology studies, and sometimes proprietary research data.

When you ask a synthetic user what they think about your checkout flow, the system is doing a sophisticated version of autocomplete: predicting what a person matching your persona description would likely say, based on patterns in its training data.

That's not nothing. Pattern matching at internet scale surfaces real insights about how people in certain demographic categories typically think and behave.

But it is also not the same as talking to a real person, and conflating the two leads to bad decisions.

What Synthetic Users Get Right

Let's be fair. Synthetic users are genuinely useful for specific things.

Rapid hypothesis generation. Before scheduling a single research session, you can stress-test your assumptions. "Would a time-pressed nurse care about this feature?" Run ten synthetic responses. If eight of them say no in different ways, you have a signal worth investigating further. Not proof. A signal.

Early-stage prototype screening. When you have five concepts and need to narrow to two before spending real research budget, synthetic feedback can help filter obvious failures. If a synthetic user cannot understand what your concept is for, a real user probably cannot either.

Terminology and language testing. Synthetic users can identify jargon and confusing copy reasonably well, because they reflect common language patterns. "Does this label make sense to a non-technical user?" is a question they can answer with decent accuracy.

Training and education. Running junior researchers or designers through synthetic user interviews before real sessions builds question-asking skills without the cost or ethical obligation of using real participants.

Availability. Synthetic users are available at 2am on a Sunday when you need to make a decision Monday morning and there is no other option. That has real value.

What Synthetic Users Get Wrong

Here is where the honest reckoning has to happen.

They cannot surprise you. Real user research is valuable because real users do things you did not predict. They interpret interfaces in ways you never imagined. They use your product in contexts you never designed for. They get stuck in places you were certain were obvious. Synthetic users, by definition, cannot deviate from patterns in their training data. They will confirm your assumptions more than they challenge them, because they are generated from average patterns, not from the chaotic reality of individual behavior.

They are trained on the internet, not on your users. Your users are not average internet users. They are people with specific jobs, specific contexts, specific mental models, and specific relationships with your product. A synthetic "45-year-old nurse" is a statistical average of what internet content says about nurses. Your actual user might be a nurse who used your competitor's product for a decade and has specific muscle memory and biases. The AI does not know that. It cannot know that.

They cannot observe behavior. The most valuable data in UX research is not what users say. It is what users do. The hesitation before clicking. The re-read of a label three times before acting. The moment someone gives up and leaves. The workaround they developed for a problem they never thought to report. None of this is capturable by a text-generating system. You can ask a synthetic user what they would do. You cannot watch them do it.

They reflect existing biases at scale. AI systems inherit the biases in their training data. If your training data underrepresents certain communities, synthetic users from those communities will not represent them accurately either. Using synthetic users as a substitute for research with underrepresented groups is not just methodologically weak: it actively risks reinforcing existing exclusions.

They hallucinate insights. This is the subtle and dangerous one. Synthetic users generate plausible-sounding feedback. They use research language. They sound like real participants. But they are predicting what a participant would say, not reporting what they actually experienced. A synthetic user can tell you with complete confidence that users would find a feature confusing, even if every real user you tested it with found it intuitive. The confidence of the output has no relationship to its accuracy.

The Real Problem with Using Them as a Replacement

Research skeptics sometimes say: "But our current research is already flawed. At least this is faster."

That is not a defense of synthetic users. It is an argument for doing better research.

The goal of user research is to reduce decision-making uncertainty. If your research process is already producing biased or misleading signals, adding a faster and more scalable version of those biased signals does not solve the problem. It makes it worse, because the volume of flawed data creates an illusion of confidence.

Teams that use synthetic users as their primary research method tend to build products that work fine in theory and fail in practice. They have all the data they need to feel confident. They have none of the data they need to actually be right.

What Actually Works Alongside AI Research Tools

The right framework is not "synthetic users or real research." It is sequencing them correctly.

Use synthetic users to: