Choosing an AI Agent Testing Approach: What UK Product Teams Actually Need to Decide

Most UK product teams reach a point where their test suite starts lying to them. Coverage metrics look healthy. Pipelines pass. And yet bugs still reach users, edge cases go undetected, and the team quietly loses confidence in what automated checks are actually catching. The instinct is to go looking for better testing tools for automation, and the market is happy to oblige with an overwhelming number of options.

But the real problem is rarely the tool. It is the decision that precedes the tool selection.

This post is not a roundup of the latest platforms. It is a structured decision framework designed to help product teams choose the right testing approach for their specific context, whether that is scripted automation, AI-assisted testing, or full AI agent simulation. Working through four practical decision criteria, including failure mode priorities, product velocity, user composition, and team maturity, you will find a clear mapping that connects your actual situation to the approach most likely to serve it. The post also covers where the UK regulatory landscape is heading, and why getting these criteria right now will matter considerably more than which tool you happen to license.

The moment the test suite stops being useful

The suite is green. Three users have reported the same broken journey in the past fortnight. No test caught it.

A product team automated their checkout and account flows eighteen months ago. Solid coverage, consistent CI runs, a suite that passed every deploy. Then they shipped an AI-assisted recommendations layer. The feature behaves differently on every run: the scripted assertions check for outputs that no longer have fixed correct values, real failures pass undetected, and the first signal of each problem is a user report rather than a red build.

This is not a tooling failure. The scripts are doing exactly what they were written to do. The product changed in a way that made the testing approach structurally inadequate, and the suite did not know to object.

The received wisdom is to fix this by adding more automation. More tests, more coverage, more scripts. The actual question is whether scripted automation is even the right category of approach for what this product now does. Adding more of the wrong thing does not close the gap; it makes the suite more expensive to maintain while catching the same failures it always caught and missing the new ones entirely.

Three approaches address this gap, scripted automation, AI-assisted testing, and AI agent simulation, and each catches different failure modes.

Choosing between them without first naming what you are trying to catch is how teams end up running expensive suites that miss the only failures that matter.

Three approaches: what they actually are

Scripted automation is a deterministic set of instructions an automated runner executes step by step against fixed assertions. Give it a selector, a sequence, and an expected value: it checks whether reality matches. It is reliable precisely because it is inflexible. That inflexibility is a feature when behaviour is predictable, and a liability when it is not.

AI-assisted testing uses a model to generate, maintain, or adapt test cases as the product changes. The assertions remain largely human-authored; the model handles the scaffolding, reducing the overhead of keeping scripts aligned with a moving UI. This is a meaningfully different thing from AI agent testing. The approach is still fundamentally scripted: a model writes the scaffolding, but the runner still follows instructions toward a fixed expected outcome. Teams sometimes conflate the two because both involve a model. The distinction matters for what each one can actually catch.

AI agent simulation sends an autonomous agent through the product with a goal rather than a script. The agent decides its own path, navigates by intent rather than by selector, and surfaces routes that no one scripted in advance. It behaves more like an actual user, or like a downstream AI system accessing your product on a user's behalf. Stunt Double's platform operates in this mode: agents simulate how real users and AI agents experience a product, covering discovery, information retrieval, task completion, and delegated access. As products increasingly interact with AI callers, not just human ones, this surface matters. What conversational AI actually does to your website covers why that shift changes what breaks first.

These three are not a maturity ladder where each replaces the last. They catch different failure modes. A team may need all three, or only one. The sections that follow map the criteria that determine which combination is right for a given product and team.

Decision criterion one: what failure modes are you actually trying to catch

The first criterion is not about tooling preference. It is about what kind of wrong you are trying to find.

Scripted automation finds deterministic failures. A button that no longer submits a form. A price that renders as null. A redirect that broke after a deploy. These are binary: the assertion either passes or it does not. Automation testing services built on this model work well where the product surface is stable and the expected output can be written as a single correct answer.

AI-assisted testing finds a different problem. When a UI changes frequently, the scripts break not because the product is wrong but because the assertions are stale. The failure is in the test suite, not the product. An AI tester that regenerates or adapts assertions reduces that maintenance burden without changing the underlying model: you are still asserting a fixed expected state; you are just spending less time keeping those assertions current.

AI agent simulation finds emergent and path-dependent failures. A conversational interface that gives contradictory answers depending on how the user arrived at it. A recommendation engine that surfaces inappropriate content under a plausible but untested prompt sequence. A checkout flow that an agent completes in a way that bypasses intended friction. These failures do not exist at a single selector or a single assertion point. They only appear when something traverses the product as an actual actor would. Understanding what an AI generator does when it visits your product is directly relevant here: the visiting actor does not follow your script.

Hallucination, prompt injection, and output drift sit in a distinct failure category that scripted tests cannot reach: they are probabilistic and context-dependent. An agent running the same task ten times with slight variation will surface these. A script run once with fixed inputs will not.

The practical decision rule: if your product includes any generative output, any LLM-backed feature, or any interface that changes its response based on prior context, scripted automation alone will not catch your most important failures. Name the failure modes first, then select the approach.

Decision criterion two: how fast is your product surface actually changing

Once you have named your failure modes, the second question is mechanical: how often does the product surface those modes change?

Rate of change is the variable that most directly predicts scripted automation debt. A product shipping one UI release per quarter can maintain a scripted suite with a modest QA investment. Research on automated test suite maintenance in industry confirms that verification and validation activities consume 20 to 50 per cent of total software development costs, and that frequent, incremental maintenance is less costly than periodic, high-effort rework. That finding assumes a relatively stable codebase. It does not account for a product shipping multiple times per week, or one where an LLM generates interface elements dynamically: in those conditions, the team spends more time fixing broken scripts than running meaningful tests.

The threshold is not a precise deploy count. It is the point at which the test suite stops being trusted. When engineers start skipping the test run because it is always red for the wrong reasons, the approach has failed regardless of tooling. That moment is easy to miss because the suite still looks like an asset on a project plan.

A useful proxy: if your scripted suite requires more than one day of maintenance work per sprint, the approach is mismatched to the rate of change. Quantify this before evaluating any new automation testing service. The number is usually available in commit history; teams rarely look.

AI-assisted testing reduces this debt by handling the script maintenance layer. The assertions still need human oversight, but the scaffolding regenerates as the UI shifts. For teams whose surface is changing faster than their scripted suite can follow, this is the appropriate upgrade path, not a wholesale change of approach.

AI agent simulation is less sensitive to surface change because the agent navigates by goal rather than by selector. If the button moves, the agent finds it. This matters particularly as AI-generated code accelerates shipping cycles and UI iteration outpaces anything a selector-based script can track. Agent-based approaches require clear task definitions and calibration of what a passing outcome looks like, but they absorb UI movement without breaking.

Decision criterion three: does your product have non-human users

Rate of change tells you how fragile your test suite is. This third criterion asks a different question: who, or what, is actually using your product.

Most testing frameworks were built with a human as the implicit actor. The selector paths, the interaction sequences, the assertion models: all of these assume a person navigating with intent, motor behaviour, and a roughly predictable decision pattern. That assumption no longer holds for a growing share of production traffic.

Products that expose APIs consumed by AI agents, products that sit inside LLM-backed copilot workflows, and products integrated with voice assistants are now being accessed by non-human actors that behave differently in material ways. They do not scroll to orient themselves. They retry on ambiguous states rather than abandoning. They interpret a 200 response with an error message in the body differently from how a human reads it. An edge case for a human user can be the default path for an agent.

A scripted test suite built around human interaction will not surface these differences. The paths are different. The failure modes are different. A delegated access scenario, where an AI agent acts on behalf of a user to complete a task, can fail silently in ways that only become visible when the agent actually runs the flow in production. No selector-based assertion catches that.

This is not a niche concern. The proportion of web traffic generated by automated systems has grown materially, and AI agents account for a rising share of that automation.

Stunt Double's platform addresses this directly: it tests across discovery, information retrieval, task completion, and delegated access, including scenarios where the actor is an AI system rather than a human. This is the same gap that makes AI-native development tools miss certain failure classes: the tool was not built to simulate the actual caller.

The decision rule is narrow. If any user, customer, or integration partner sends an AI agent into your product, your testing approach needs an agent-perspective simulation. No scripted automation suite, however thorough, will cover that surface.

Decision criterion four: where is your team on the AI maturity curve

The non-human actor question is about who is using your product. This criterion is about whether your team has the infrastructure to act on what testing finds.

Research across large enterprise samples consistently identifies a progression through stages of AI maturity, from initial experimentation through isolated implementation, enterprise scaling, and embedded governance. The testing approach appropriate at the earliest stage is structurally different from the one appropriate at the most advanced stage, because the product surface, the team's interpretive capacity, and the risk exposure are all different.

Stages one and two: build the foundation first. At these stages the priority is baseline scripted coverage. If a team has no app automation testing in place, starting with AI agent simulation is premature. Agent runs will surface findings, but without established coverage discipline there is no reference point for what good looks like, and the results are harder to triage and act on. Scripted automation instils that discipline. It is the floor, not a legacy constraint.

Stage three: layer, do not replace. At this stage the team has scripted coverage and is beginning to ship AI-backed features. The scripted suite remains useful for the deterministic core but is no longer sufficient across the whole surface. The appropriate move is to add AI-assisted testing for maintenance efficiency where the UI is changing fast, and introduce agent simulation specifically for the AI-backed journeys, not as a wholesale replacement for everything that came before.

Stage four: governance becomes the frame. At this stage the product may include generative features, AI-to-AI interaction surfaces, and complex delegation scenarios. The NIST AI Risk Management Framework, extended with the Generative AI Profile, is clear that the testing approach should reflect the organisation's risk tolerance and lifecycle stage rather than a universal best practice. For context on how AI compresses and reshapes the product life cycle at every stage, the implications run deeper than testing alone.

A practical self-assessment requires only three numbers: AI-backed features currently in production, user journeys that produce non-deterministic output, and integration points where a downstream AI system consumes your product. Those three numbers locate you on the curve more accurately than any survey.

Mapping the decision: a framework for UK product teams

Those four criteria, taken together, form a matrix rather than a sequence. Where you sit determines which approach earns its cost.

Low AI feature density, stable surfaces, no non-human user exposure: scripted automation with good coverage discipline is the right answer. Switching to agent simulation because it is the current topic of conversation adds cost without adding signal. The suite is not broken. Leave it alone.

Moderate AI feature density, moderate rate of change, some API surface exposed to automated callers: evaluate AI-assisted testing for maintenance efficiency, then introduce targeted agent simulation for the AI-backed journeys specifically. This is the most common position for UK product teams shipping in 2026, and the move is additive, not wholesale. If you are in this position and wondering what that agent-facing surface actually looks like in practice, what flow AI is, and why product teams need to test for it gives the concrete grounding.

High AI feature density, rapid iteration, generative output in production, non-human user exposure: agent simulation becomes a first-class method, not a supplementary one. Scripted automation still handles the deterministic core. The shift is that agent simulation results become the primary signal. Running them as a secondary check, after the scripted suite has already passed, inverts the priority.

Risk tolerance modifies all three positions. A product operating in UK financial services or healthcare carries a higher cost of failure than a marketing tool. That asymmetry shifts the recommendation toward more comprehensive agent simulation even at lower AI maturity levels, because a missed failure in a regulated context is disproportionately expensive to remediate.

NIST is direct on this: profiles should reflect organisational risk tolerance and resources rather than prescribe universal solutions. A framework that ignores risk tolerance will systematically undertest high-stakes products. That is the more dangerous error.

The UK regulatory context: what product teams should track

Risk tolerance, as the previous section established, is not a single dial. For UK product teams in regulated sectors, it is partly set by the regulator.

The UK's approach to AI governance, as of 2026, is sector-led rather than statute-led. There is no single binding AI Act. Instead, the FCA applies existing governance, operational resilience, and consumer protection frameworks to AI systems in financial services. The ICO applies data protection obligations. The MHRA applies its existing device and software frameworks to healthcare AI. The compliance question is not "does this product satisfy the AI Act" but "which regulator owns our sector, and what does our existing obligations framework require of AI systems now."

That matters for testing approach selection. In a sector-led model, auditability of your testing process becomes part of the compliance record. A testing run that produces results you cannot attribute to a specific agent behaviour, sequence, or state is harder to defend under an operational resilience review than one that surfaces named artefacts. Stunt Double's platform outputs exactly this: what the agent did, what it encountered, and where the experience broke. That structure is not incidental. It is the kind of documented evidence that a second category of AI detector makes legible: not checking text for AI origin, but confirming whether AI agents can actually navigate and use a product as intended.

NIST's AI Risk Management Framework and its Generative AI Profile are US instruments, but UK and international governance bodies reference them as a cross-sectoral baseline. For any team shipping LLM-backed features, the Generative AI Profile names failure modes that sector regulators will increasingly expect teams to have tested against. UK teams building for EU markets carry an additional obligation: products that fall under high-risk categories in the EU AI Act face mandatory conformity assessment requirements, creating a parallel compliance track for cross-border products.

Teams outside regulated sectors should still track this direction. Governance frameworks become expectations before they become requirements. Regulatory scrutiny of AI systems in financial services is already examining whether existing supervisory frameworks are sufficient as AI systems become more autonomous. Building a documented, maturity-based testing approach now costs less than retrofitting one after the landscape formalises.

Where scripted automation still wins

Regulatory obligations reinforce a point the earlier criteria have been circling: some flows must not deviate. That is exactly where scripted automation belongs.

Forms, payment flows, authentication sequences, data validation: these are binary surfaces. The correct output is a single known value, the assertion is either true or false, and a well-maintained scripted suite produces regression confidence on every deploy. There is no ambiguity to simulate. The script is the right instrument precisely because there is only one acceptable outcome.

Compliance-critical flows extend this logic further. A regulated onboarding sequence where each step must occur in a fixed order is not a candidate for agent simulation as the primary method. Deviation from the sequence is itself the failure. What you need is a deterministic runner that follows the prescribed path and fails loudly if any step is skipped or reordered. App automation testing was built for exactly this.

The mistake most teams make is not over-relying on scripted automation for these deterministic surfaces. That reliance is correct. The mistake is failing to notice when a new feature has crossed into non-deterministic territory and continuing to apply scripted assertions as if the boundary has not moved. The suite stays green. The product accumulates failures the scripts are structurally incapable of seeing.

A practical signal: if you cannot write the expected output of a feature as a single correct answer, scripted assertion-based testing is not the right primary instrument for that feature. It may still be useful for the surrounding plumbing, the database writes, the API response codes, the redirect logic. But the feature itself needs a different approach.

Mature teams structure their testing investment so that scripted automation handles the deterministic core and agent simulation handles the non-deterministic surface. The budget question is not which one to choose: it is what proportion of your product surface is now non-deterministic, and whether your testing investment reflects that split.

How to run a maturity-based testing review in one sprint

Knowing your proportion of non-deterministic surface is the input. Running the review that produces that number is the work. Five steps, one sprint.

Step one: inventory your product surface. List every user-facing journey. Categorise each as deterministic (same input, same output, every time) or non-deterministic (output varies by context, model state, or prior interaction). Count both columns. The ratio is your first decision input and the clearest signal you have before touching any tooling question.

Step two: audit your current test suite against that inventory. Where is scripted coverage concentrated? Almost certainly on the deterministic side. Identify which non-deterministic journeys have scripted tests applied to them anyway. Those are your false-confidence entries: tests that run green while the actual failure mode goes undetected. The gap between what is scripted and what needs agent simulation is the finding this step produces.

Step three: map failure mode exposure. For each non-deterministic journey, name the relevant failure modes: hallucination, output drift, prompt injection, path-dependent inconsistency, non-human agent incompatibility. Then score each journey by the severity of its failure consequence, not by likelihood. A low-probability hallucination in a medical information flow scores higher than a frequent but low-consequence drift in a product description. This scoring is your risk register for approach selection, and it is the document a regulator or risk committee will ask for first.

Step four: assess AI maturity using three proxy counts. How many AI-backed features are in production? How many user journeys produce non-deterministic output? How many integration points exist where a downstream AI system consumes your product? These three numbers locate your team on the four-stage maturity model more reliably than any survey instrument.

Step five: match approach to surface and maturity. Scripted automation covers the deterministic core. AI-assisted testing addresses journeys where maintenance overhead is the primary cost. Agent simulation covers non-deterministic journeys with material failure risk. Write down the rationale for each allocation. That document is the artefact a stakeholder or regulator will ask for. Produce it while the decisions are fresh.

The decision before the tool

Once the sprint review is complete, the artefact in front of you is not a shopping list. It is a set of answers that make the tool question answerable.

Product teams reliably arrive at the same starting point: which testing tool should we use. The question that actually determines whether their testing works is different: which failure modes are we trying to catch, and does our current approach have any chance of catching them? Those are not the same question. Confusing them is how teams invest in new tooling and find themselves in exactly the same position six months later.

The four criteria in this piece, failure mode priority, rate of product change, non-human user exposure, and team maturity, are not a checklist to file away. They are the mechanism by which approach selection becomes defensible. Get those four answers right and the tool choice follows directly. Get them wrong and no tool will close the gap, because the gap is not a tooling problem.

This framework will not tell you which tool to buy. It will tell you what you are actually deciding, so that the tool choice is the last step rather than the first. The tool follows the approach. The approach follows the criteria. The criteria follow the inventory.

Start with the inventory. The rest follows.

Conclusion

UK product teams have a narrow window to close the gap between green test suites and real user experience before regulatory scrutiny makes that gap costly.

Audit your testing inventory this sprint. Map it against the four criteria. Let the approach follow the evidence, not the other way around. The clarity is already there; you just have to look for it.