User testing platforms: what they measure, and what they miss

A participant completes your onboarding in three minutes. You keep the recording, the task completion rate and the score they gave the flow at the end. Two weeks later a customer writes in to say they never found the export button.

That gap is the thing to design around.

What a user testing platform actually does

A user testing platform recruits people, puts a task in front of them and stores what happened. You write the task, the platform supplies a participant, and you read the recording or the numbers afterwards.

Maze and UserTesting are the two names most product teams meet first, and both sell the same core thing: a real person, a real attempt, a record you can point at. The differences sit in recruitment, moderation, and how much of the analysis is done for you.

What the panel model is good at

Unmoderated studies give you volume. Run one task with twenty people and you see where they stop. Moderated sessions give you the reason they stopped, in their words, while you watch.

That pairing has held up for years because it answers a question no dashboard can: why did this person give up here?

Where the panel model stops

The received wisdom is that more sessions mean more certainty. The number of sessions is not the constraint. The participant is.

So what happens when the visitor is not a person?

An agent lands on the page, reads the markup, clicks through and leaves in four seconds. Your analytics file that as a bounce with no engagement, in the same bucket as a human who did the same thing because the page was broken. The two sessions look identical in the report.

There is a second limit. A panel tells you about the version you tested. Ship on Tuesday and the recording is a document about last week.

Running the same task twice

Agent runs close that gap. You write the task once as a checklist, run it in a real browser, and keep the screenshot from each step. Then you run it again after the next release and compare the two.

When a step that passed on Monday fails on Thursday, you have a date, a screenshot and a step number. That is a bug report, not a research finding.

The Stunt Double Index is the public version of this: agents run the same set of tasks against real sites and the results are ranked. Read it if you want to see the shape of an agent run before you build one.

What to look for in a user testing platform

• Can you rerun a task without recruiting a new panel?

• Does each run produce an artefact you can attach to a ticket?

• Can it report what the agent saw, not only that it failed?

• Does it tell a human bounce apart from an agent session?

Most platforms answer the first two questions well and skip the last two. That is fine if your only visitor is a person.

The part most teams skip

Usability work usually stops when the study ends. The findings go into a doc, the doc gets read once, and the next release ships without anyone re-running the task.

Agent runs are cheap enough to keep. A checklist you rerun every release is a regression suite for experience, and it catches the failures that unit tests were never written for.

A panel tells you what a person did once. A checklist tells you what your product does every time you ship.