Connect an agent to a browser through a tool server and you can ask it, in English, to search a shop for a laptop and add it to the cart. It opens a real browser, finds the elements, clicks them, checks the result, and hands back a summary. No selectors, no waits, no page objects.
The demos are genuinely impressive and the framing around them is wrong. It gets sold as testing without specification — describe the outcome, skip the code. That is not what happens, and you can see it in any of these demos if you count the lines.
Look at the prompt
Here is the shape of a working prompt for a search-and-add-to-cart flow, condensed but not exaggerated:
Goal: verify a user can search for a product and add it to the cart.
Precondition: user is on the homepage.
1. Navigate to the shop.
2. Enter "MacBook Air" into the search bar.
3. Submit the search.
4. Validate that results display products matching "MacBook Air".
5. Hover the first product named "MacBook Air", click the Cart icon,
then click "View Cart" as soon as the popup appears.
6. Confirm the cart count is "1" and that the message
"Products marked with *** are not available in the desired
quantity or not in stock!" is displayed.
7. Take a screenshot.
Expected: results relevant to the keyword, correct cart count,
message displayed on the cart page.Count what is in there. The interaction sequence, in order. The disambiguation rule for picking among several results. The exact interaction — hover then click, not click. A timing hint for the popup. The expected string, character for character, including the asterisks.
That is a test specification. It is written in prose rather than TypeScript, and it is longer than the Playwright code that comes out the other end. The thing that got removed was not the specifying. It was the syntax.
Which is worth being clear-eyed about, because the value proposition changes completely once you see it. You are not skipping the hard part. You are moving it into a less precise notation.
What actually gets saved
There is a real saving here and it is worth naming precisely, because it is the part that decides whether this belongs in your workflow.
The agent resolves locators at runtime. It loads the page, looks at what is there, and works out that the cart control for a given product is the icon that appears on hover inside that product’s tile. You did not write [data-product] >> nth=0 >> .cart-btn, and you did not spend twenty minutes in devtools discovering that the button only exists in the DOM after a hover event.
That is genuinely the tedious part of browser automation, and it is the part that breaks most often. Someone renames a class, the tile markup changes, a wrapper div appears — the selector dies and the test goes red for a reason that has nothing to do with the product being broken.
So the honest framing is not “testing without code”. It is automation that resolves locators at runtime instead of at authoring time. That is a smaller claim and a much more useful one, because it tells you exactly where it helps and exactly where it hurts.
Where it hurts: the same prompt runs differently
Runtime resolution is the feature, and it is also the problem, because runtime resolution means the decision is remade on every run.
Two runs of the same prompt against the same page can pick different elements. The model might match the product by its accessible name once and by its position in the grid the next time. Both are reasonable. They fail differently: the first breaks when the copy changes, the second breaks when sort order changes. You did not choose either, and you cannot read your test to find out which one you got.
For a suite you run on every commit, that is disqualifying on its own. A test whose behaviour is decided fresh each run cannot serve as a regression check, because when it goes red you cannot tell whether the product changed, the page changed, or the model just picked differently this morning. That is not flakiness in the usual sense — the environment is fine and there is no race to fix. It is non-determinism, which you bound rather than eliminate, and a per-commit gate is exactly the place where you cannot afford to bound it.
There is also a cost floor. Every step is a model call with a page snapshot attached, so a flow that takes a compiled test two seconds takes the agent a minute and costs real money. Multiply by suite size and by commits per day.
Where it genuinely wins
Two places, and neither of them is the per-commit suite.
Generating the first draft of a test. Let the agent walk the flow once, watch what it does, and have it emit the Playwright code. Now you have a real test with real locators, which you read, fix, and commit. The agent did the devtools archaeology; you own the result. The artefact that lands in the repository is then deterministic code rather than a prompt, so everything in the previous section stops applying.
Exploration, which is the thing a script cannot do at all. Ask the agent to complete a checkout without telling it how, and watch where it hesitates, backtracks, or picks the wrong control. It has no memory of your app and no idea what you intended, which makes its confusion a signal. A script cannot be confused. That is the one capability here with no equivalent in traditional automation, and it maps onto the part of testing that was never automatable: does this interface make sense to someone encountering it cold.
The second one is undersold because it produces no code and therefore looks like it produced nothing. A transcript of an agent failing to find your “continue as guest” option is a usability finding, and it arrived in ninety seconds.
Reviewing what comes out
If you use it for the first draft, the generated code needs the same review as any generated code, with one addition specific to this route.
The usual checks apply: is the assertion meaningful or does it just confirm the page loaded, are the waits real conditions rather than sleeps, is the test independent of the ones around it. Those are the same questions you would ask about any code an agent produced.
The addition is locator quality. The agent optimised for working right now on this page, which is a different objective from surviving the next six months of refactors. It will happily emit a positional selector or a text match on a marketing string, because both worked when it looked. Rewriting those to test-ids or roles is the review step that decides whether the test is an asset or a future ticket.
The practical position
Treat it as an authoring and exploration tool, not an execution model.
- Use it to explore an unfamiliar flow and to surface where the interface is unclear. This is the strongest use and the least discussed.
- Use it to draft tests you then read, harden, and commit as ordinary code.
- Do not put a prompt in the commit path and call it a test. Non-determinism at the gate costs you the ability to trust red.
- Budget for it. Per-step model calls with page snapshots are not free at suite scale.
The name suggests it removes the specification. It does not — the prompt above is a specification, and a fairly demanding one. What it removes is the selector archaeology, which is worth removing, and it is a large enough win that it does not need the bigger claim.