The Agent Bug That Breaks Nothing and Costs the Most

Most bugs announce themselves. A stack trace, a 500, a red test, a page that will not load. You find out because something stopped.

Multi-agent systems have a class that does the opposite. Nothing stops. Every component behaves exactly as written, every log line is unremarkable, and the system produces correct answers the whole time. You find out four weeks later, from the invoice.

I hit one building a supervisor — a router that hands work to specialist agents. Below is that bug, three of its cousins, and the property they share: the failure mode is that nothing fails.

Working code: stages/s03_router/ at tag stage-03. Runs offline, no API key. 38 checks, 20 of them on failure modes, and every number below is measured.

The supervisor is not new architecture

One sentence carries the whole pattern:

A supervisor is the same agent you already have, with other agents as its tools.

Same loop, same registry, same model call. The only difference is that behind the tool name sits an agent with a narrow toolset rather than a function.

The routing itself is 49 executable lines. There is a dict of specialists, one model call to pick one, and a while with a counter. Nothing in it is clever, and that is the point — if you have never written the routing by hand, the framework version looks like magic instead of like add_node.

What is not simple is what happens around it. Three things appear the moment work crosses from the supervisor to a specialist, and each of them fails silently.

Silent failure 1 — the revision loop with no counter

A supervisor can disagree with a specialist’s answer and send the work back. That is the whole reason you would build one instead of a cheap classifier.

Now remove the limit on how many times it may do that:

if state.revisions >= state.revision_limit:   # ← delete this
    state.finish_reason = "revision_limit"
    return state

Against a scripted fake model, a test goes red immediately: the script runs out of replies. That red is an artefact of testing. Here is what the same deletion does in production:

no exception       the model answers as many times as it is asked
no log line        every cycle looks like ordinary work
no timeout         unless you set one

The system keeps working. Answers keep arriving. Nothing degrades in any way a dashboard can show.

I measured the cost on the smallest possible case — a specialist that does not call the model at all, so only the judging step costs anything:

 limit   model calls   revisions   finish reason      cost/run
     0             2           0   revision_limit     $0.004
     1             3           1   revision_limit     $0.006
     2             4           2   revision_limit     $0.008
     5             7           5   revision_limit     $0.014
    10            12          10   revision_limit     $0.024

Calls = limit + 2, and that is the floor. Make the specialist an agent with its own loop and each revision costs as many calls as that loop takes. Now delete the limit and there is no last column at all, because the run does not finish.

This is the same guard as a step limit in a single agent loop, in the same position — before the action, not after. If you have a step limit and no revision limit, you have half a guard.

Silent failure 2 — the access level that does not survive the handoff

A specialist receives a task, not a person. Whatever the supervisor knew and did not pass along does not exist downstream. The first casualty is authorization.

Give the knowledge specialist a hardcoded access level instead of reading it from the run state — pin it to public — and four checks go red. Look at which four:

graph: the operator gets the internal document through the same route
FAILURE · specialists: the operator receives what they are allowed to
FAILURE · graph: request text cannot raise the access level
e2e · demo runs offline and shows five scenes

Not one of them is a leak test. Pinning to public is fail-safe: everyone sees less than they should. Nothing leaked, so nothing that tests for leaks could fire.

What broke is the mirror: a support operator stopped seeing the internal refund thresholds, and will quote a customer the wrong number just as quietly as before.

Now pin it to internal instead. Five checks red, and the set barely overlaps:

FAILURE · graph: internal document does not reach a shopper through the handoff
FAILURE · graph: the permitted answer does arrive
FAILURE · graph: request text cannot raise the access level
FAILURE · specialists: the knowledge specialist reads access from the state
FAILURE · specialists: the operator receives what they are allowed to

One line, two opposite defects, two nearly disjoint sets of red. Neither set covers the other — which is why there are three authorization checks and not one:

the forbidden did not arrive     the test everyone writes
the permitted did arrive         the test that catches the mirror
the request text changed nothing the test that catches the abuse

The access level lives in the run state rather than being passed at each handoff, and nothing may write it. Passing it explicitly works right up until someone adds a fourth specialist and forgets the line — and forgetting is cheap while the consequence is silent.

Silent failure 3 — the state schema as a free dict

The run state is read and written by every node. The easy version is a dict: adding a field costs one line and nobody declares anything. Most examples look like that, and it is why most examples do not show the expensive decision in the room.

When adding a field costs one line, nobody asks what the field actually costs.

It costs however many nodes come to depend on it, and none of them will announce the dependency. Six months later you cannot rename it or remove it, because there is no way to find out who reads it.

Declaring the schema makes that cost visible at the moment you pay it, and __slots__ gives you the contract for free:

class State:
    __slots__ = DECLARED   # reading or writing anything else is an error

That also buys a distinction worth stating out loud, because the two events look identical in a log and demand opposite handling:

a specialist raises            a fact about the world  -> becomes a step result, run continues
a node reads a missing field   a broken contract       -> the run stops, naming the field

The warehouse can be down; the graph must survive that. A node reading an undeclared field means the contract is broken, and continuing on an empty value is worse than stopping. Get this backwards — catch everything — and a broken contract quietly redresses itself as an environment event.

The checks that were green while the door stood open

Everything above was tested. Then two independent reviewers went through it, and two mutations left the entire suite green.

Neither was a bug in the code. Both were checks guarding the wrong thing.

The first. The check that the access level cannot be overwritten was written like this:

for name in sorted(FROZEN):
    ...assert the write is refused...

Empty FROZEN and the loop body never executes. The suite stays green while every field becomes writable, including the one the whole design exists to protect. The check iterated the very constant it guarded.

Worse: the lesson told readers to make exactly that mutation, while the exercises told them to make a different one that did go red. Two instructions for one exercise with opposite results, and both were written by someone who believed the property was covered.

The second. Removing the re-raise that lets a contract error escape the graph changed nothing either — because no check ever drove an undeclared field read through the graph. There was a check on the state object directly, and it proved the state object works. It proved nothing about the system.

Both are the same shape: a check with the right verdict and too weak a claim. Mutation testing does not catch these, because they do go red when you break the thing they nominally test.

A green suite that means less than it says

One more, and it is the one I would look for in your CI today.

The route comparison against a framework implementation returned early when the framework was not installed — and its verdict printed as ok. So “matched” and “not checked” looked identical in the output. Meanwhile CI installed only the base dependencies, so the comparison ran nowhere at all, and the pipeline was green.

The fix has to be both halves, and the second alone is worse than nothing:

a third state in the runner    NOT VERIFIED, counted separately, never printed as ok
a CI job that installs extras  and fails if anything is still unverified

Add the third state without the job and you have documented, precisely and permanently, that a guard never runs.

There is a trap on the way there, too. The obvious generalisation — treat any import failure as “not verified” — would have made CI go green on a genuinely broken build. A package missing because it is an optional extra and a package missing because the core dependency list is wrong produce the same traceback. Only the dependency table tells them apart, so that is what the runner reads.

I learned this the direct way: CI failed twice on ModuleNotFoundError: numpy, because numpy sat in a stage extra while the shared layer imported it unconditionally. Locally everything was green — numpy had been installed at some point along the way. Now a twenty-line script runs the suite with the optional packages blocked through a sitecustomize on PYTHONPATH, so the block survives the per-module subprocesses.

Do you even need a supervisor

Usually not, and that deserves saying before anyone builds one from this article.

Three verdicts, not two, and the middle one is the one people skip:

A classifier is a cheap branch pick with no revision loop. Most systems built as supervisors needed exactly that: the routing, not the loop. And the last row is the most common case and the most common mistake — five tools in one domain need a registry, not a graph.

The ordering matters as much as the rows. Structural constraints first, then cost, and size last — size is the weakest argument and the one people reach for first.

Try breaking it

git clone https://github.com/AZANIR/agentic-ai && cd agentic-ai
git checkout stage-03
pip install -e ".[dev]"
python -m stages.s03_router.run
python -m stages.s03_router.check

Then, one at a time — each has a measured expected result, so you can tell “I broke it correctly” from “I broke something else”:

1. Make the revision limit unreachable. Then run python -m stages.s03_router.solutions.exercise_2_revision_cost and read the last column.
2. Pin the specialist’s access level to "public", then to "internal". Two opposite defects, two nearly disjoint sets of red.
3. Empty the frozen-fields set. This is the mutation that used to pass.
4. Delete the route validation so the graph trusts whatever node name the model returns.

The exercise numbers are not written by hand — python scripts/mutate.py s03 --expect applies each mutation, counts the red checks and fails if the prose disagrees with the run. That script exists because the first version of the page claimed nine red for exercise 3 when the truth was three: the mutation as written did not compile, and the nine came from a crashed specialist rather than a changed access level.

What this does not cover

Routing quality is not proven here. On a scripted fake the route is correct by construction; a real model routes differently and sometimes worse. That is a quantity you measure, and measuring it is a different piece of work.

The hand-rolled graph is not production-shaped either — 49 lines demonstrate the mechanics and know nothing about parallel branches, checkpointing, or recovery. Read it to understand what a framework does for you, then use the framework.

But the four silent failures above are not framework-specific and no library removes them. They are properties of handing work from one agent to another, and each one is invisible in exactly the way that matters:

A bug that breaks nothing is not a small bug. It is a bug with no discovery mechanism.