Synthetic Users: What AI Participants Can and Cannot Tell You

Nielsen Norman Group took three studies it had already run with real participants and reran them with AI stand-ins, testing both ChatGPT and the Synthetic Users product¹. One of the three covered online training. In NN/g's data, real learners commonly start courses and never finish them. The synthetic learners reported completing everything.

That gap is the subject of this guide. A synthetic user is an AI-generated profile built to mimic a user group, producing research-shaped findings without any real person involved¹. The pitch is easy to see: answers in minutes instead of weeks, near-zero cost per participant, simulated access to groups you struggle to recruit. The published comparisons keep finding the same problems in the output.

Why teams reach for them

Recruiting enterprise admins can take time and require incentives. A synthetic panel can respond much sooner, which makes it tempting when research time and budget are tight.

Groups such as compliance officers, ICU nurses, and CFOs at mid-market firms can be hard to recruit. A simulation may be tempting when you cannot reach them.

Synthetic transcripts, quotes, and personas can look like the outputs of a real study. Label them clearly so stakeholders know how they were produced.

What the comparisons found

MeasuringU reviewed 12 papers that compared synthetic users with human participants across four research contexts. They tagged 9 findings encouraging and 14 discouraging, summarizing the field as "the results aren't universally bad, but they definitely aren't great"².

The distribution of those findings matters more than the totals. On matched variance, the review found zero encouraging findings and three discouraging ones: synthetic data shows artificially low variability. On qualitative depth, the same pattern, zero encouraging and three discouraging². These findings raise concerns about using synthetic responses to understand differences between people or the detail of their experiences.

NN/g's three-study comparison illustrates the problem.¹ The chatbots wanted to please, a tendency the researchers name directly as sycophancy, and did not model human behavior well. The synthetic users' stated values were too shallow to act on. They "seem to care about everything," which tells a PM nothing about what to prioritize, and probing only produced more vagueness. Their past experiences were invented, because a model has never used your product. Whatever it offers as memory is confabulation.

The failures also run in the opposite direction. Jeff Sauro and colleagues tested ChatGPT on tree testing, the task of locating items in a navigation hierarchy, and concluded it was not an acceptable replacement for real users because it vastly outperformed most real people³. The concept and task studies show different risks. Human data provides a check on whether the simulated responses resemble actual behavior.

Two more results from the academic side. Rafikova and Voronin (2026) found synthetic users can match the direction of human attitudinal trends while missing the deeper structure underneath them². And Wang, Morgenstern, and Dickerson, writing in Nature Machine Intelligence in 2025, found that LLMs substituting for human participants "can harmfully misportray and flatten identity groups"². If you are simulating users from a demographic you do not belong to, the simulation is not a neutral stand-in.

Five failure modes

Daniel Russell highlights the difficulty of representing social context, such as collaboration and shared devices, through simulated individuals.⁴ He also notes the depth gap. Real participants articulate thought processes, tell stories, and bring life context that a generated profile cannot supply.

The trap: concept validation

NN/g warns against concept validation because synthetic responses often favor the proposed idea.¹ Sycophancy is one risk: a model may agree with the way a concept is presented, making an enthusiastic description seem validated⁵.

NN/g warns that teams can become comfortable relying on synthetic responses and stop studying real users. If those responses later prove wrong, the experience can also damage trust in subsequent research¹.

A model can suggest an unexpected possibility, but that possibility is not an observed customer experience. Use it to broaden the questions you investigate.⁵

Where synthetic users earn a place

Use synthetic participants to prepare for research rather than replace participant data⁵. NN/g and User Evaluation describe uses such as the following¹⁵:

1. Pilot your discussion guide. Run your interview questions against a synthetic participant before the first real session. This may help flag confusing phrasing or leading questions before the first session.
2. Explore an unfamiliar user group. Build proto-personas for a segment you have never studied, explicitly labeled as drafts to be corrected by real research.
3. Generate hypotheses. Ask the model what objections, edge cases, and unmet needs a given segment might have. Each answer is a candidate, never a finding.
4. Run pre-mortems and stimulus pre-tests. "This launch failed in six months, tell me why" is a prompt a synthetic user handles well, because you want plausible failure stories rather than truth about any individual.
5. List possible edge cases. Ask the model for scenarios your guide could cover, then check them against published research or real participants.

Recruit real participants for the qualitative study, using the synthetic round to improve the questions. For a quantitative test, calculate the required sample before starting. An underpowered test may leave the hypothesis unresolved. The A/B testing guide covers the experiment design.

Using AI to analyze data from real participants is a different practice. It can introduce errors such as fabricated quotes, so outputs still need checking against the source. That workflow is covered in AI for user research synthesis. This article concerns models used in place of participants.

FAQ

What is a synthetic user? An AI-generated profile that mimics a target user group and produces artificial research findings, such as interview answers or survey responses, without any real participant involved¹. Vendors sell dedicated products for this, and teams also improvise the same thing with ChatGPT or Claude.

How accurate are synthetic users compared to real participants? Results vary across studies and tasks. A review of 12 published comparisons tagged 9 findings encouraging and 14 discouraging, with zero encouraging findings on matched variance and zero on qualitative depth². Synthetic users can track the direction of human attitudes while missing the structure underneath, and they both overrate concepts and over-perform on tasks²³.

Can I use synthetic users to validate a product concept? Do not use synthetic approval as validation. NN/g found overly favorable feedback in its evaluation.¹ Test demand with real participants before acting on that feedback.

What are synthetic users actually good for? Work that happens before research: piloting a discussion guide, drafting proto-personas for an unfamiliar segment, generating hypotheses, running pre-mortems, and enumerating edge cases¹⁵. Treat the outputs as possibilities to investigate with people.

Is this the same as using AI to synthesize my user interviews? No. Synthesis applies AI to data real people produced, and the main risk (fabricated quotes) is checkable against the source transcripts. Synthetic users replace the people, so there is no source to check against. See AI for user research synthesis for the first practice.

Sources

Footnotes

1. Synthetic Users: If, When, and How to Use AI-Generated "Research" (Nielsen Norman Group) ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹

2. A Review of Experiments with Synthetic Users (MeasuringU) ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

3. Can ChatGPT Replace Humans in a Tree Test? (MeasuringU) ↩ ↩²

4. The Challenges of Synthetic Users in UX Research, Daniel M. Russell (ACM Interactions) ↩ ↩²

5. Synthetic Users vs Real Participants: What the 2026 Research Actually Shows (User Evaluation) ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶