When I started thinking about my own QA agent, the idea seemed fairly simple at first: the agent should generate a test, run it, and show the result. But the deeper I went, the more obvious it became: a real QA process isn’t linear.
Testing almost never looks like “wrote it — ran it — done.” More often it’s a cycle. Write a test, run it, get an error, realize the problem is in the selector, fix it, run again, see that the test passes but checks the wrong thing, add an assertion, run again, think about a negative scenario, add data, check cleanup.
That’s why the idea of an autonomous QA agent quickly ran into architecture. If the system is to behave like an engineering assistant, it needs not a simple chain of actions, but a controlled cycle with the ability to go back.
That’s how I arrived at the idea that Cairn should be built around a state graph.
Why a linear workflow isn’t enough
Many AI automations start with a pipeline: get a task, form a prompt, generate a response, save the result. For simple scenarios that’s enough. But QA doesn’t fit that scheme.
In testing, the result of each step affects the next. If the test doesn’t compile, you need to go back to generation. If it compiles but fails at runtime, you need to pass the log to the agent. If it passes, you need to evaluate whether it’s a false positive. If the evaluation is poor, you shouldn’t save the test — you should improve it.
This means the system must have memory of state: which test was generated, what errors occurred, what attempts were already made, what the judge said, which criteria weren’t met, how many iterations have run.
In a plain linear approach this logic quickly turns into chaos of callbacks, conditions, intermediate variables, and “glue” code. For an autonomous agent that’s dangerous, because it’s very easy to lose transparency: something seems to have happened, but why the system made that decision — unclear.
LangGraph as the foundation of the cycle
LangGraph fits this task well because it lets you describe a process as a set of nodes and transitions. Not one big agent that “somehow figures it out on its own,” but separate steps with clear responsibility.
For Cairn this can look like:
Observe — analyze the repository, configs, test structure, stack.
Orient — determine which skills are needed for the task.
Plan — form a test plan or list of scenarios.
Generate — create test code.
Execute — run the test and collect the result.
Analyze — read errors, logs, trace, stdout.
Repair — return to generation with a specific reason for the failure.
Judge — evaluate test quality if it technically passes.
Persist — save the test if it’s genuinely useful.
In this scheme the LLM isn’t responsible for all the magic at once. It works inside specific nodes, while the workflow is driven by the graph. That makes the system more engineering-oriented. You can explicitly see where the agent is, why it went back, at which step the problem arose, and what state is passed forward.
Why Langfuse belongs here
The next problem you hit with autonomous agents is transparency. If the agent did something wrong, it’s not enough to see the final error. You need to understand the whole path.
What prompt did the model get? What context was there at that moment? Which tool did it call? How many tokens were spent? What did the executor return? Why did the judge reject the test? At which iteration did the agent start repeating the same mistake?
Without observability an autonomous agent quickly becomes a black box. It may sometimes give a good result, but it’s hard to improve systematically.
That’s why integration with Langfuse matters for Cairn. The idea isn’t just to store pretty logs. The idea is to make the agent’s behavior visible.
Each task can be a trace. Each graph node — a span. Each model call — a generation. Judge evaluations — scores. Then you can look not only at the final test, but at the whole process of its birth: from the first idea to the last decision.
This is especially important for optimization. For example, if you see the generator spending too many tokens on irrelevant context, you can improve the skill. If the judge often returns the same remark, you can change the rubric or generation prompt. If the executor returns an incomplete log, you can fix the tool — not blame the model.
Why you need an LLM judge
When a test fails, things are relatively simple: there’s a technical error to fix. But when a test passes, a harder question arises: is this a good test?
You can’t answer that with pass/fail status alone. Pass only means the test didn’t crash. It doesn’t guarantee the test checks the right behavior.
That’s where LLM-as-a-judge comes in. But not an abstract judge that rates “answer quality” — a specialized judge for QA code.
Such a judge should look at a test through practical criteria:
Coverage. Does the test actually check the stated business logic? Are important edge cases present?
Reliability. Does the test use fragile selectors? Does it handle async correctly? Does it depend on random timing?
Isolation. Does the test create its own state? Does it clean up data afterward? Can it run independently?
Value. Does this test reduce real product risk, or just add another green line to the report?
This ties directly to the idea of eval as a truth mechanism. The judge in Cairn shouldn’t be a decorative score. Its job isn’t to say “8 out of 10.” Its job is to help the system decide: keep this test, improve it, or reject it.
A cycle in practice
Suppose we want to create checks for a password reset flow.
Cairn analyzes the project and sees the frontend is tested with Playwright, and an API can be used to prepare a user. It activates the relevant skills and forms a plan: successful reset, non-existent email, expired token, weak password, token reuse.
Next the generator creates the first test. The executor runs it. The test fails because the page uses different button text than the model expected. The Analyze node returns the exact log. The graph moves the process back to Generate. The second version uses a more stable locator. The test passes.
But the cycle doesn’t end there. The judge looks at the code and sees the test only checks the “Password changed” message — but doesn’t verify the user can actually log in with the new password. Weak test. It passes, but doesn’t prove the core business logic.
Cairn returns to generation with the judge’s comment. The next version adds login with the new password and a check that the old password no longer works. Only then can the test be considered a useful artifact.
Why this matters for the future of AI-assisted QA
I think the main mistake in many AI approaches to testing is stopping at generation. Generating a test is the easy part. Much harder is building a system that can doubt its own result.
In that sense Cairn shouldn’t be just a “bot that writes tests.” It should be a system that goes through the cycle: plan, generate, run, analyze, evaluate, improve. That cycle is what makes it useful not as a demo, but as a tool for real engineering work.
And here it’s important that all three parts — LangGraph, Langfuse, and the LLM judge — play different roles.
LangGraph provides a controlled process.
Langfuse provides transparency.
The LLM judge provides a quality mechanism.
Together they form the foundation for an autonomous QA cycle where AI doesn’t just respond, but works inside a system of checks.
Conclusion
After my earlier work on eval, it stopped being just an evaluation technique for me. It became a way of thinking: don’t take the result on faith — build a mechanism that moves you closer to the truth.
Cairn was born from exactly that logic. If AI helps move faster from idea to code, you need another agent layer that helps move from code to proof.
LangGraph, Langfuse, and a custom LLM judge aren’t just a set of trendy tools. They’re an attempt to build a QA system that doesn’t stop at “the test passed,” but asks: “Does this test actually prove anything?”
And as AI development gets faster, those are exactly the questions that matter most.