An agent runs fine for a dozen steps, then starts making choices that do not follow from anything. It calls a tool that has nothing to do with the task. It contradicts an instruction it followed correctly ten steps ago. It rehashes an approach that already failed.
The reflex is to blame the model, and the reflex is usually wrong. What changed is not the model. It is what the model is looking at.
That much is now well covered. The part that gets less attention is the part a QA engineer actually needs: these failures are distinguishable. There are four of them, they produce different symptoms, and the fix for one makes another worse. Treating “the agent got confused” as a single diagnosis is how teams end up compressing context that was never too long, or adding tools to an agent that already had too many.
This is a testing problem before it is a design problem. So the useful question is not “how do I engineer context well” — it is how do I know which of the four is happening right now.
Why the window degrades before it fills
One piece of background matters, because it sets the threshold for everything below.
A context window does not behave like a disk that works perfectly until full. Chroma’s context rot study evaluated eighteen frontier models and found performance degrading continuously as input length grew, well below the stated limit — a gradient, not a cliff. A model advertising 200K tokens can measurably lose accuracy at 50K.
There is a second, sharper effect on top of that. Attention across a long context is not uniform: material at the beginning and end is recalled reliably, material in the middle is not. For an agent, that has a specific consequence — your system prompt starts at the top, and after forty thousand tokens of tool output it is no longer at the top of anything that matters. The instructions did not get deleted. They got buried.
The practitioner rule of thumb that has settled out of this: quality starts slipping somewhere around 40-60% of the window. Treat that as your budget, not the advertised number.
The four failure modes
Drew Breunig’s taxonomy is the useful one, and it holds up. What follows is that taxonomy plus the part it does not cover: what each failure looks like from the outside, and what to check.
Poisoning — a bad fact that keeps getting re-read
A hallucination or a wrong tool result enters the context and stays there. The agent reads it on every subsequent step and builds on it. Because agents iterate on their own output, one bad step compounds instead of washing out.
Symptom: the agent is internally consistent and externally wrong. Every step follows logically from the last. Nothing looks broken.
How to detect it: this is the hardest of the four, because the transcript looks clean. Work backwards: find the first step where output diverged from reality and check what entered the context immediately before it. In practice that means you need tool outputs logged separately from the model’s reading of them, so you can tell “the tool returned garbage” from “the tool was fine, the model misread it.”
What actually fixes it: validating tool output at the boundary, before it enters the context, the same instinct as validating input at a trust boundary in ordinary code. And after the agent recovers from an error, drop the failed attempts. Ten steps of dead-end debugging left visible is ten steps of wrong reasoning the model will consult again.
Distraction — the model stops thinking and starts pattern-matching
As history grows, the model leans harder on what it has recently seen and less on what it learned in training. Instead of forming a new plan, it repeats the shape of previous actions.
Symptom: repetitive, plausible, unproductive. The agent keeps doing variations of what it already did. It rarely errors — it just stops making progress.
How to detect it: this one is measurable. Track tool-call sequences per run and look for repeating n-grams. A loop of the same three calls with slightly different arguments is distraction, and you can alert on it without a human reading transcripts.
What actually fixes it: summarising older history, applied before you feel you need to. Note this is the one failure where a bigger context window makes things worse rather than better: more room means more history to over-index on.
Confusion — too many tools, not too many tokens
Superfluous content in the context gets picked up and acted on. The classic case is tool definitions: an agent with too many tools starts calling ones irrelevant to the task.
The number that makes this concrete comes from a reported benchmark run: a quantized Llama 3.1 8B failed on GeoEngine when given all 46 available tools, and passed when given 19. The context was well within its window limit either way. The tools were not too many to hold. They were too many to reason about.
Symptom: wrong tool selected, and it is the wrong kind of wrong — not a near-miss, but a tool from an unrelated domain.
How to detect it: log which tools were available at each call alongside which was chosen, then look at selection accuracy as a function of tool-set size. If accuracy drops as the set grows while context length stays flat, it is confusion and not rot. This is the single most useful instrument on the list, because it separates two failures that look identical in a transcript.
What actually fixes it: fewer tools in context per step. Semantic retrieval over tool descriptions — surfacing only what this step needs — is the scalable version, and the reported gains are large: one paper measured tool-selection accuracy going from roughly 14% to 43% while roughly halving prompt tokens.
That fix has a cost, and it is the interesting part of this whole topic. See below.
Clash — two sources of truth, no ordering between them
New information contradicts something already in context. The system prompt says one thing; a retrieved document says another. The model cannot reconcile them, so behavior becomes inconsistent — sometimes following one, sometimes the other, sometimes neither.
Symptom: the same input produces different behavior across runs. This is the failure most likely to be misfiled as flakiness.
How to detect it: re-run the same scenario several times and diff the tool-call sequences. Genuine nondeterminism is noisy in small ways; a clash produces runs that diverge cleanly into two or three distinct branches. Any QA engineer who has chased a flaky test knows the difference between “jittery” and “bimodal” — this is bimodal.
What actually fixes it: an explicit authority ordering. System prompt beats retrieved facts beats conversation history, stated somewhere the model can see it, with structured sections so it can tell which source a claim came from.
This is the failure mode where a structured knowledge layer earns its cost. When I looked at why RAG alone is not enough, the argument was that vector search returns similar fragments rather than authoritative ones. Clash is what that looks like at runtime: the retrieval worked, and the agent still cannot tell which of two similar answers outranks the other.
Telling them apart
| Failure | Symptom | Cheapest signal | Wrong fix |
|---|---|---|---|
| Poisoning | Consistent and wrong | Tool output logged separately from model reading | Compression — summarising carries the bad fact forward; it has to be removed, not shrunk |
| Distraction | Repetitive, no progress | Repeating n-grams in tool-call sequence | More context window |
| Confusion | Unrelated tool called | Selection accuracy vs. tool-set size | Compression — the problem is not length |
| Clash | Bimodal across identical runs | Diff tool sequences across N reruns | Retries — both branches are “successful” |
The right-hand column is the reason to bother distinguishing them. Compression is the reflexive fix for context problems and it is the wrong answer for two of the four.
The tension nobody resolves
Here is where the standard advice pulls against itself, and where you have to choose rather than follow.
Fixing confusion means changing the tool set per step. Load only what this step needs.
Doing that destroys your prompt cache. Inference providers cache the computed key-value representations of your prompt prefix. Keep the prefix byte-identical and the cached portion is reused; change anything early and it is recomputed from scratch. Tool definitions sit near the top. Rotating them per step invalidates everything after them, every call.
The saving at stake is real. On Sonnet-tier pricing, uncached input runs $3 per million tokens against roughly $0.30 per million for a cache read — a 10× difference, on an agent making dozens of calls per task.
What the write-up I took this from left out, and what changes the arithmetic: cache writes are not free either. A write costs about 1.25× the base input rate for the standard TTL, and around 2× for the longer one. So a prefix you invalidate and rewrite every step does not merely lose the 10× discount — it costs more than never caching at all. The naive “just keep the prefix stable” advice is right, but the penalty for getting it wrong is worse than it sounds.
Two ways out, and they suit different scales:
- Tool masking. Keep every definition in the context, stable and cached, and mark the irrelevant ones unavailable for this phase. The prefix never changes. Works while the full set is small enough that the model can still reason over it — which puts you back under the confusion threshold, so this is a fix with a ceiling.
- Retrieval over tool descriptions. Genuinely fetch only what is needed. Scales to large tool sets, and pays the cache cost every step.
The general principle underneath: stable content at the top, volatile content at the bottom. System prompt and tool definitions first; conversation history, current step, and agent state last. That ordering is free and most agent harnesses get it wrong by accident.
What this changes about testing an agent
Three things follow that are not obvious from the design-side framing.
Context state belongs in your test output. A failing agent run that records only the final answer is unactionable, because all four failures produce “wrong answer.” Record token count at each step, which tools were in context, and where each fact entered. That is the difference between a bug report and a shrug.
Some agent flakiness is not flakiness. A test that fails one run in four may be hitting a clash, and clashes are deterministic given the same context, and only look random because the retrieval order varies. Before you mark a scenario flaky and add a retry, re-run it and check whether the failures cluster into distinct branches.
Your eval set needs long-context cases. A suite where every case is a short interaction will not catch rot, distraction, or anything else that emerges at step 15. When I argued that an eval is a truth mechanism rather than a grade, this is a case in point: an eval that only exercises short paths gives a confident number about a region of behavior you do not actually ship in.
That last one is worth being concrete about. If your agent runs 20-step tasks in production and your longest eval case is 4 steps, you have measured a different system.
Where to start
Pick the instrument, not the strategy. In order of return on effort:
- Log context size per step. One number, and it tells you whether you are anywhere near the 40-60% band where quality starts sliding. Most teams do not know this number.
- Log tools-available alongside tool-chosen. This is what separates confusion from everything else, and it costs a line of logging.
- Re-run failing scenarios three times and diff the tool sequences. Free, and it converts “flaky” into either “clash” or “genuinely flaky.”
Only then reach for the strategies: writing state out, retrieving selectively, compressing, isolating into sub-agents. They all work. But applying a strategy before you know which failure you have is how a team spends a sprint compressing context that was never too long, on an agent that had too many tools.
The framing that has served me best: an agent’s context is a system under test, not a prompt. It has state, that state has failure modes, and the failure modes have signatures. Instrument it like anything else you would be paged for.