Before writing any prompts, one team did something unusual: they wrote down what a test automation engineer actually does across a sprint. Not what the job description says. What consumes the hours.
Five categories came out, and three of them account for roughly eighty percent of the work:
- New test development — requirements, design, code, run, ship.
- Test maintenance — updating tests that broke or got disabled because the product moved.
- Failure investigation — reading logs and deciding whether this is a product defect, an infrastructure problem, or a flake.
Look at which one everybody builds agents for.
It is the first one, every time. Ticket goes in, pull request comes out. And the first one is arguably the worst of the three to start with.
Why authoring wins the attention and loses the argument
Authoring demos beautifully. The input is a ticket, the output is a diff, and both fit on a slide. It is also the part of the job people find most identifiably skilled, so automating it feels like the impressive claim.
The trouble is what happens to the output. A generated test enters your repository and stays there for years. If it is subtly wrong — asserts the wrong thing, uses a brittle locator, duplicates coverage you already had — you have not saved work, you have created maintenance, which is category two, which was already large.
Authoring is worth doing, and the constraints that make it survivable are mostly about keeping the diff reviewable. But it is the category where a mistake compounds, and it is the one where a human still has to read every line carefully. The ceiling on the saving is the review.
Triage has the opposite shape
Failure investigation is where a QA engineer’s morning goes. Forty tests are red. Most of them are the same root cause. Somebody has to open each one, read the log, and sort it into a bucket.
Everything about that task suits an agent, and in ways authoring does not:
The output is a classification, not an artefact. Product defect, environment problem, flake, or genuinely unknown. Nothing lands in the repository. A wrong classification costs one wasted look, and it is corrected by the human who looks. Compare that to a wrong test, which lives on.
It is genuinely mechanical. Read the stack trace, check whether the same failure appears across unrelated tests, check whether it passed on retry, check whether the environment had an incident. These are lookups, and they are the reason the task is tedious rather than interesting.
The answer is checkable in seconds. A human confirming or overturning a triage verdict takes a glance. This is the property that makes an eval trivial to build — a hundred historical failures with known causes, and the score is exact agreement, no judge model whose own accuracy you would then have to measure.
The alternative is not a human doing it well. Under a pile of red, people pattern-match and rerun. The realistic comparison is not agent-versus-careful-engineer, it is agent-versus-rushed-engineer, and that is a much easier bar.
Maintenance sits in between. A test that broke because a selector changed is a narrow, verifiable fix. A test that broke because the feature genuinely changed needs judgement about what the assertion should now be, and that is authoring wearing a disguise.
Route the model by stage, not by system
The other thing worth taking from a build like this is that a twelve-stage pipeline does not want one model.
The stages differ enormously in what they demand. Estimating a task by decomposing it into known units is close to arithmetic. Classifying a failure against a fixed taxonomy is classification. Deciding whether an implementation satisfies a requirement is the expensive kind of judgement, and so is reviewing a diff for what it forgot.
| Stage | What it really is | Tier |
|---|---|---|
| Estimation | Decomposition against known units | Cheap |
| Failure classification | Bucketing with lookups | Cheap |
| Log summarisation | Extraction | Cheap |
| Selector repair | Pattern match plus verification | Mid |
| Test design | Judgement about coverage | Expensive |
| Self-review | Judgement about what is missing | Expensive |
The pattern that holds up is expensive-model-decides, cheap-model-fetches, and the mistake in both directions is common: one team runs the whole chain on the strongest model and pays for arithmetic at reasoning prices, another runs it all on the cheapest and gets a review stage that approves everything.
Worth revisiting periodically, too. Model choice per stage tends to get made once, during the build, and then never looked at again while both the models and the prices move.
The state file is what makes a long chain possible
One structural detail from this build is easy to skim past and is load-bearing.
Each task gets a history file. Every stage writes to it: the branch that was created, the estimate, the tracker issue, what happened at each step. Later stages read what earlier stages recorded.
That is the mechanism that lets a twelve-stage chain exist at all. Without it, stage nine either carries everything from stages one through eight in context — which degrades as it grows, well before any limit — or it starts blind. The file is an external memory that keeps each stage’s working context small, and it has the useful side effect of being readable by a human when something goes wrong, which a context window is not.
It is also the honest answer to why delegation is worth its cost here: each stage consumes a lot and writes down a little, and the little is what the next stage needs.
Where to start on Monday
If you have one week to point an agent at your automation workload:
- Do the decomposition first, on paper. Where do the hours actually go? The answer is frequently not what the team believes, and it is the only input that matters.
- Point it at failure triage. Feed it a run’s worth of red tests plus logs, ask for a classification and a one-line reason per failure. No write access to anything.
- Score it against history. A hundred past failures with known causes. You will have a number by Wednesday, and it is an exact-match number.
- Leave the verdict advisory. It sorts and explains; a human decides. This is what keeps a wrong classification cheap.
- Only then look at authoring, and treat the review as the constraint that sets the ceiling.
The reason to start here is not that triage is more valuable in the abstract. It is that triage is where a mistake costs a glance, and authoring is where a mistake costs a year of maintenance — and the second is where everyone begins.