The usual pitch for spec-driven development goes like this: stop typing one-line prompts at a coding agent. Write what the feature should do, then a technical plan, then tasks, and let the agent execute them. It is a good pitch. A spec removes the guessing that makes agents invent architecture.
It leaves one question open that matters to anyone who tests software. When the agent finishes the last task and reports success, what has been verified?
Not as much as the tidy artifact trail suggests.
What the workflow does and does not check
GitHub’s Spec Kit is the best-known toolkit for this style. Its reference describes nine agent commands: constitution, specify, clarify, plan, checklist, tasks, analyze, implement and converge (Spec Kit reference). The exact invocation depends on your agent, so you will see both /speckit.plan and /speckit-plan in the wild.
Read what each one does to the question of testing:
clarifyasks up to five questions about underspecified areas and writes the answers back into the spec. It improves the spec. It tests nothing.checklistgenerates a requirements-quality checklist. The docs call it “unit tests for your requirements”, and a checked item means the requirement is well formed, not that the feature works.tasksproduces a dependency-orderedtasks.md. Tests appear inside a story’s phase when requested. Ask for none, get none.analyzeis a read-only consistency check across spec, plan and tasks. It will flag a task with no matching requirement. It will not flag a requirement with no test.implementruns the tasks.convergecompares the code against spec, plan and tasks and appends missing tasks.
Every one of these checks the artifacts against each other or the code against the artifacts. None of them executes your acceptance criteria. So there is a gap, and it is the same one an old-fashioned requirements document had: the requirement says what must hold, and nothing forces a test to exist for it.
If you have read the piece on specs for agents, the premise carries over: a spec is a test plan in disguise. Here the point is mechanical. A plan nobody runs is a wish.
Number the criteria, tag the tests, fail on the gap
The fix is small and boring. Give every acceptance criterion an identifier, put the identifier on the test that proves it, and make CI fail when a criterion has no test.
Take a feature small enough to hold in your head: notification preferences for employees. The spec’s acceptance section:
## Acceptance criteria
- AC-1: An employee can turn email notifications off, and the setting persists.
- AC-2: An employee can turn push notifications off, and the setting persists.
- AC-3: An existing employee who never set preferences gets email on and push on.
- AC-4: Updating preferences does not change any other employee field.
- AC-5: A preference key the system does not know is rejected with 400.Playwright can tag a test with the tag option. The tests, against a tiny in-memory service I wrote for this article:
import { test, expect } from "@playwright/test";
test("email can be turned off and persists", { tag: "@AC-1" }, async ({ request }) => {
await request.put("/employees/e1/preferences", { data: { email: false } });
expect(await (await request.get("/employees/e1/preferences")).json())
.toMatchObject({ email: false });
});
test("existing employee gets defaults", { tag: "@AC-3" }, async ({ request }) => {
// e2 never sets anything in this file
expect(await (await request.get("/employees/e2/preferences")).json())
.toEqual({ email: true, push: true });
});
test("updating preferences leaves other fields alone", { tag: "@AC-4" }, async ({ request }) => {
await request.put("/employees/e2/preferences", { data: { email: false } });
expect(await (await request.get("/employees/e2")).json())
.toMatchObject({ name: "Bo", email: "bo@example.com" });
});(AC-2 follows the same shape as AC-1 and is omitted here.)
Then the gate. It reads the criteria out of the spec, asks Playwright for its test list as JSON, and compares the two:
import { readFileSync } from "node:fs";
import { execFileSync } from "node:child_process";
const acs = [...readFileSync("spec.md", "utf8").matchAll(/^- (AC-\d+):/gm)].map((m) => m[1]);
const list = JSON.parse(execFileSync("npx", ["playwright", "test", "--list", "--reporter=json"],
{ encoding: "utf8", shell: true }));
const specs = (s) => [...(s.specs ?? []), ...(s.suites ?? []).flatMap(specs)];
const tagged = new Set(list.suites.flatMap(specs).flatMap((t) => t.tags));
const uncovered = acs.filter((a) => !tagged.has(a));
const unknown = [...tagged].filter((t) => !acs.includes(t));
uncovered.forEach((a) => console.log(`UNCOVERED ${a}`));
unknown.forEach((a) => console.log(`UNKNOWN ${a} (tag with no criterion)`));
console.log(`${acs.length - uncovered.length}/${acs.length} criteria covered`);
process.exit(uncovered.length || unknown.length ? 1 : 0);Run against the files above (Playwright 1.64.0, Node 24):
UNCOVERED AC-5
4/5 criteria covered
exit 1Two things worth noticing. The check did not need to understand the tests, only that a tag exists. I left AC-5’s test out on purpose, so the gate has something to find. In a real project that is exactly the failure the artifact trail hides, since the spec, the plan and the task list can all be internally consistent while AC-5 has no check at all.
The UNKNOWN line catches the reverse: a test tagged @AC-9 after the spec was edited and renumbered.
The criterion earns its keep when the code is wrong
A trace check only proves a test exists. The reason to want one is what it does to a defective implementation. I added a switch to the service that reproduces a mistake generated code makes easily: the preferences update rebuilds the whole employee record from the new preferences and drops everything else. This is a planted bug that mimics the pattern, not output from an agent.
correct service:
4 passed
BUG=1:
- "email": "bo@example.com",
- "name": "Bo",
> 20 | expect(await (await request.get("/employees/e2")).json())
.toMatchObject({ name: "Bo", email: "bo@example.com" });
1 failed
3 passedAC-1 to AC-3 stayed green because they only look at the preferences. AC-4 is the one that notices a name has vanished. This is why “does not change any other field” belongs in the spec as its own criterion. Without it the natural test set passes and the corruption ships.
What goes into the spec to make this work
- Write criteria as observable outcomes. “The setting persists” can become a GET after a PUT. “Is user friendly” cannot.
- One behaviour per criterion. If a criterion needs “and” twice, the tag points at a test that proves half of it.
- Ask for tests when you run
tasks. The docs say they appear when requested, so request them, and name the tag convention in the project’s constitution so every task follows it. - Run the gate in CI beside the tests. A missing criterion should fail a build, not a review comment.
- Use
clarifyon the gaps the gate finds. AC-5 exists because someone worried about unknown keys. If you cannot say what the system should return, the spec is incomplete, and that is aclarifyquestion rather than a test to guess at.
Where this breaks
A tag is not a test of the criterion. The gate accepts any tagged test. My first version of the AC-3 test only asserted that the response had email and push keys, which is true no matter what the values are. It would have passed a service returning both as false. I caught it on reread and rewrote it to check values. Coverage by tag is a floor; the quality of the assertion is still a review problem.
The agent that writes the code often writes the tests. Tests produced in the same pass as the implementation tend to confirm what was built. Ask for the tests from the spec before the plan exists, or from a separate pass that has not seen the code. A test pyramid still applies to agent-written code, and a test derived from the code cannot fail it.
Shared state in my example. The tests rely on running in order in one file against one server, with e1 and e2 kept apart. A real suite would isolate state per test.
Identifiers rot. Insert a criterion in the middle of a spec and every later number moves unless you keep IDs stable. Treat them like public identifiers: never reuse one, never renumber.
This is one toolkit, one stack. The pattern is independent of Spec Kit: any workflow that produces numbered criteria can use it. The Playwright details are what I ran, not a general rule.
What to take from it
A spec-driven workflow gives an agent better context. It does not give you evidence. The cheapest piece of evidence is a number on every criterion and a test that carries it, with a build that fails when a number has nothing behind it.
Start with the next spec you write: number the acceptance criteria, ask for tagged tests, and see how many criteria turn up uncovered on the first run. In my five-line example it was one, because I had left it out. In a real spec you will not know which one until the gate tells you.