Self-Healing Tests Repair the Symptom

The arc is consistent enough to set your watch by.

Week one, a team points a model at a feature and gets tests. Week two, they get a lot of tests. By week four the pipeline is slower, red is common, and nobody quite believes the suite any more.

Nothing in that story is about the tests being wrong. Most of them pass, most of the time. The problem is that generation is the cheap half of the job and the expensive half was never assigned to anyone.

Generated tests carry no opinion about what is fragile

Write twenty tests by hand and you carry things you never type out. Why this one exists. Which parts of the page are stable and which get redesigned every quarter. What can change without anyone caring.

A model has none of that unless you built it in. So you get more tests and the same understanding, and the ratio between those two is what actually decides whether a suite survives.

The decay shows up in a few reliable places:

• Selectors pinned too tightly. An id, a position, a class that existed on the day the test was written.
• Timing baked into assertions. A wait that matched one machine on one afternoon.
• Exact-match assertions. Full object comparisons and snapshot-shaped checks, which look thorough and are brittle.

That third one deserves its own moment, because it fails in the most expensive direction.

Your API adds an optional field. Nothing has broken. A human-written test asserted the two things it cared about and sails through; the generated one compared the whole object and goes red. Now the team is looking at a failure that means nothing, and they have learned something they cannot unlearn: red does not necessarily mean broken.

Where it goes from there is predictable. Either failures start getting ignored, or somebody weakens the assertions until they stop firing, and coverage becomes a number rather than a claim.

The proposed fix, and what it costs

The tidy answer doing the rounds is a healer. Three agents: one plans the test, one runs it, one watches for failures and repairs them. A locator stops matching, the healer finds the element that moved, rewrites the selector, and the test is green again without anyone being paged.

For the failure it is aimed at, it works. It also removes the thing you most needed.

A test going red because a locator no longer matches is a notification that the interface changed. That is not a defect in the test. It is the only mechanism you have that connects a change in the product to a human being told about it. Heal it silently and the notification is gone, permanently, and what remains is a suite that quietly tracks whatever the application happens to do.

Two failures look identical from inside the runner and could not be more different:

The runner cannot tell these apart. It sees an element that used to match and no longer does. Which means an automatic healer resolves the ambiguity by guessing, and it guesses in the direction of green — the direction that generates no work and no signal.

That is the same trap as rerunning until a test passes, with better engineering behind it. A gate that gets bypassed by habit is worse than no gate, and a gate that bypasses itself is worse still.

Healing is a proposal, not a merge

There is a version of this that keeps the benefit. The difference is entirely in what the healer is permitted to do with its answer.

Let it diagnose. Let it locate the element, work out what the selector should now be, and write that down. Then have it open a change for review rather than apply one, and let a red build stay red until a person looks.

What you get is the tedious part automated — the devtools archaeology, the “what does this element look like now” — with the decision still owned. Ninety seconds of review instead of thirty minutes of investigation, and no test silently rewrites its own meaning.

The cheap version of this needs no agents at all. A locator that heals twice in a month is not a healing problem, it is a test asserting on something that was never stable, and the fix is a test-id.

Refactoring is the loop nobody runs

The deeper gap is that senior engineers do not only write tests. They keep rewriting the ones that exist — pulling out duplication, extracting the setup that four files repeat, dragging the suite along as the architecture moves.

Generation does none of that, and it is not close. Six weeks of unsupervised generation produces a suite that reads like five teams with no shared conventions wrote it, which in a sense is what happened.

So the practices that keep a generated suite alive are mostly not about generation:

Assert on contracts, not on shapes. Check the fields the behaviour depends on. A response that grew a field did not break.

The distinction is small in code and large in maintenance:

// Shape: fails when anything anywhere in the payload moves
expect(res.body).toEqual({ id: 7, status: "shipped", carrier: "DHL" });

// Contract: fails when the thing under test is wrong
expect(res.body).toMatchObject({ status: "shipped" });
expect(res.body.id).toBe(7);

The first version is what generation reaches for, because the whole response was sitting there when the test was written and including it looks more rigorous. It is the assertion most likely to go red for a reason nobody cares about, which makes it the one most likely to teach a team that red is noise.

Give the model the conventions before it writes. Which helpers exist, which selectors are sanctioned, where setup belongs. This is the same restraint that keeps an agent’s pull request reviewable: the useful instruction is usually about staying inside existing patterns, not about writing good code in general.

Budget for deletion. A generated suite grows faster than a written one, so the hour that removes tests matters more, not less.

Watch the flake rate as a rate. Fifteen flaky tests in eight hundred is not five in a hundred with more zeroes. Each one costs attention on every run, and attention is the fixed budget everything here competes for.

The question to ask about any healer

Before wiring one in, one question settles it: when this repairs something, does a human find out?

If yes, you have automated an investigation. Keep it.

If no, you have built something that makes red rarer without making the product better, and the number it improves is the one you were using to decide whether to ship.

Generation made tests cheap. It did not make them free, and the part it did not touch — knowing which failures mean something — is the part the whole exercise was for.