JetBrains recently ran a controlled benchmark on Caveman, a popular agent skill that compresses model replies into short, stripped-down prose. The README claims roughly 65% output-token savings. The skill has tens of thousands of GitHub stars. Influencers treated it like a breakthrough.
The measured result on real agent work: 8.5% output-token reduction. Task quality did not move in any meaningful direction. On total spend, the skill arm ended up 11.6% more expensive than baseline, mostly because one long-context outlier erased the expected savings.
That gap between marketing and measurement is not unique to Caveman. It is the default pattern for a whole category of AI productivity tools.
What JetBrains actually tested
The JetBrains write-up is worth reading in full because the methodology is careful.
| Parameter | Value |
|---|---|
| Harness | Harbor 0.17, Docker-sandboxed trials |
| Agent | Claude Code 2.1.200, headless |
| Model | Claude Sonnet 5, low reasoning effort |
| Benchmark | SkillsBench: 82 paired tasks with auto-graded verifiers |
| Comparison | Stock Claude Code vs. Caveman forced on every reply |
Two details matter for interpreting the numbers.
Forced activation is the optimistic case. Caveman normally triggers on phrases like “caveman mode.” In the benchmark it was enabled on every response. Real-world savings would likely be equal or lower.
Agent output is mostly not prose. Code, diffs, tool calls, and exact error strings dominate token usage. Caveman correctly leaves those byte-exact. Only the narration between tool calls gets compressed, and there is not much of it on multi-step agent tasks.
That explains why a skill advertised at 65% lands near single digits on coding work.
The three findings
Finding 1: token savings were small. Across 82 clean pairs, output tokens dropped from about 592k to 542k. Useful, but nowhere near the headline number, which was measured on chat-style answers rather than agent sessions.
Finding 2: quality did not degrade. Paired task scores were statistically flat. Eight tasks scored higher with the skill, ten lower, sixty-four tied. Average score gap: 0.015 on a 0-1 scale. The sign test p-value was 0.82. Caveman changed style, not outcomes.
Finding 3: cost savings were fragile. Per-task expectation pointed to roughly 10% cheaper runs. Total arm cost still inverted because one dependency-audit trial crossed the 200k long-context pricing tier and billed $8.29 vs. $0.33 in the baseline arm. Variance from a single outlier can erase a modest average gain.
JetBrains closed with a fair summary: use Caveman if you like the style, but do not budget around large savings on daily agentic work.
The same story, different repo
Caveman is one entry in a growing shelf of agent add-ons that promise speed, memory, or token efficiency:
- MemPalace — long-term memory for agents
- rtk — context compression for CLI workflows
- headroom — context window management
- ponytail — minimalism-first agent instructions
Some of these are thoughtfully built. Some are experiments worth trying on a side project. The problem is not that every repo is useless. The problem is the category’s sales pitch: install this skill, flip this switch, and your throughput jumps by an order of magnitude.
That pitch mirrors an older genre: the “one magic prompt that will 10x your work” thread. New packaging, same assumption that the bottleneck is wording rather than workflow.
Why this rarely becomes a platform feature
The logic is straightforward.
If a technique reliably cut agent cost by half without hurting outcomes, the platform vendor would ship it. Context management, memory, and output shaping are core economics for Claude Code, Codex, Copilot, and Cursor. A durable 30–60% win does not stay a third-party README for long.
What usually survives as an external skill is one of three things:
- Style preference — Caveman changes how the agent talks, not what it ships
- Niche workflow glue — useful in one stack, noisy in another
- Unverified marketing — star count as proof, benchmark optional
GitHub stars measure attention, not effect size. A repo can be fun, well-branded, and still fail a paired benchmark on the work you actually do.
This is where eval as a truth mechanism becomes practical. Do not ask whether a skill sounds clever. Ask whether paired runs on your tasks score the same, cost less, and finish faster. One smoke test is noise. Eighty-two paired trials start to mean something.
What actually speeds up agent work
The unglamorous answer is discipline in how you run the agent, not another compression layer on top.
These are the rules I keep in AGENTS.md and CLAUDE.md because they change outcomes more than any single skill install:
- State assumptions and tradeoffs before the first line of code. If something is unclear, read the codebase or ask. Guessing is expensive.
- Prefer the smallest correct solution. KISS and YAGNI by default. Reach for DRY and SOLID when the task has clearly outgrown a one-off script.
- Touch only what the task requires. Report unrelated bugs and tech debt after the scoped work is done. Do not widen the diff mid-task.
- Define a verifiable done criterion up front. “Looks good” is not a test plan. Work until the criterion passes.
- Stress-test the plan before implementation. When execution breaks, fix the strategy first. Patching random lines without revisiting the approach burns tokens and time.
None of this is novel software engineering. That is the point. Agent tooling amplifies whatever process you already have. A sloppy workflow with a token-saving skill is still a sloppy workflow, just with shorter sentences between tool calls.
For token hygiene specifically, boring habits beat exotic skills: fresh sessions between unrelated tasks, lean CLAUDE.md files, disabling unused MCP servers, and keeping context focused. I wrote up the practical version of that in Ocean’s 18 token hacks. Most of the gains there come from context discipline, not dialect tricks.
Read the transcript, not the hype
If you want ground truth on what your agent is doing, turn on verbose output and read it.
Claude Code terminal: /config → Output Style → Verbose
Claude desktop app: menu (three dots, top right) → Transcript → Verbose
Verbose mode shows reasoning, tool selection, and where the session spends context. You will see the same patterns the benchmarks measure: repeated file reads, scope creep, premature “done” claims, and tool calls that should have been a single grep.
That log is more useful than any star count. It tells you whether the agent understood the task, whether your instructions were tight enough, and whether the failure was model quality or process quality.
The broader lesson from AI coding as systems work applies here too. The tool matters less than the operating procedure around it. Skills are accessories. Verification, scope control, and clear completion criteria are the engine.
Conclusion
Caveman is a useful case study because the benchmark was honest and the gap was large. Advertised 65% savings became 8.5% on real agent tasks. Quality held steady. Total cost did not reliably drop.
That does not make every agent skill worthless. It does make the default posture skeptical: measure paired outcomes, watch for single-trial outliers, and separate style preferences from performance claims.
The work that compounds is less exciting than a viral GitHub repo. Tight project instructions. Verifiable done criteria. Small diffs. Reading verbose transcripts when something feels off. Re-running checks before you trust the green checkmark, the same verification mindset I described when building Cairn’s QA layer.
Agent skills will keep appearing every week. Most will be optional. Few will deserve the headline multiplier attached to them. The boring discipline was doing the heavy lifting before Caveman existed, and it will still be doing the heavy lifting after the next 85k-star repo drops.