You can run a clean usability test on an AI feature, watch five people succeed at the task, ship it, and still be wrong. Not because the test was sloppy, but because you tested one roll of the dice. The feature is probabilistic. The same input can return a different answer on the next run, and the answer your participants saw may be the best one they will ever get.

That is the problem at the center of AI UX research. Classic research validates a fixed thing: this screen, this flow, this copy. An AI feature is not fixed. It is a distribution of possible outputs, and a single happy-path session tells you almost nothing about the rest of that distribution. If your method does not account for variance, your research is theater. It looks like validation and proves nothing.

This piece is about research methods that hold up when the thing under test changes every run, and how to wire those methods back to a number you already track so the result is a decision, not a slide. It pairs with the UX patterns that drive feature adoption, which covers what you do with the findings once you have them.

What makes AI UX research different from regular UX research

The unit under test is a distribution, not a screen. That is the whole difference, and everything else follows from it.

Regular UX research assumes a deterministic product. Press the button, get the menu. The designer specified every state, so research checks whether users can find and operate those states. AI features break that assumption. As Jakob Nielsen put it, generative AI introduced intent-based outcome specification, where the user states a desired outcome and the system decides how to produce it. The system is no longer obedient. It is probabilistic, and it can produce a different result for the same request.

So a test that confirms "the user got a good answer" confirms one sample from a range. The next user, or the same user an hour later, might get a vague answer, a confidently wrong answer, or a refusal. Research that only sees the good sample will overstate quality and miss the failure modes that actually drive people to abandon the feature. Good AI UX research treats variance as the subject, not the noise.

What should you test in ai ux research

Test four things classic research usually skips. Task success on the happy path is the least useful of them.

The first column is the gap. Most AI feature research stops at row one of a normal usability script and never tests recovery, trust, or the metric. Those three are where AI features quietly fail. A feature can be usable and still erode trust, and a feature can feel impressive in a session and still move zero. If you only test what regular research tests, you learn what regular research learns, which is not enough for a feature whose output you cannot predict.

How do you run ux research on ai features

Sample the distribution, then score it against a baseline. The protocol is small and you can run it in a week.

Keep the qualitative panel small. The classic finding still holds: a first study finds about 85% of usability problems with just five participants, so five users is plenty per round. The change for AI is that you do not run each task once. You run the same task many times to see the range of outputs, then put the panel in front of a representative slice of that range, including the bad ones. Five users seeing one curated good answer is the trap. Five users seeing the real spread is research.

Here is the shape of the protocol:

AI feature research protocol (one task, one week)
 
1. Pick ONE task that maps to a metric you already track.
   metric_baseline = current value (e.g. task completion = 62%)
 
2. Sample the output distribution.
   Run the same input 20-30 times.
   Bucket results: good / vague / wrong-but-confident / refusal
   variance_score = % of runs that are "good enough to ship"
 
3. Run 5 users against a REAL slice (include 1-2 bad outputs).
   For each session, log:
     - task_success     (did they finish?)
     - caught_error     (did they notice the bad output?)
     - trust_rating     (1-5, self-reported)
     - recovery_path    (what did they do when it was wrong?)
 
4. Decide.
   keep IF variance_score is acceptable
          AND users recover from bad output
          AND projected metric_delta > 0
   else cut or rescope.

Step four is the point. The output of AI UX research is not a list of friction points. It is a keep-or-kill call backed by how the feature behaves across its real range. The second internal read, the UX patterns that drive feature adoption, covers how to act on a "keep" without over-promising to users.