What really burns tokens is sub agents. I once gave Claude Code a pretty big task, and it immediately launched 7 sub agents which burned through my budget before even one of them was finished. Tried again 5 hours later: same result.
If I let the main agent do the same task sequentially, it was no problem at all. I don't know if it's really just communication and orchestration that makes sub agents so inefficient, or if Anthropic figured that most people using sub agents pay per token on a big corporate account, so this is an easy way to make more money from tokenmaxxers.
thejazzman 3 minutes ago [-]
for subagents to be cheap/effective, you have to specify the size of those subagents; i.e. right now by default 5.6-sol spawns many 5.6-sol subagents. 5.4-mini is king and saves me tons of tokens
korrectional 23 minutes ago [-]
My opinion is that claude code uses more tokens simply because Anthropic makes more money that way and forces people into their subscriptions. This is supported by the fact that they won't let you use your sub on a different coding agent. I use pi btw.
toddmorey 15 minutes ago [-]
I thought I read somewhere that according to filings for going public, subscription revenue is tiny… like 5%.
Edit: consumer Claude subs are the 5%. I’d bet most all of CC subs lump in under enterprise.
- API & Enterprise: 75% to 85% of total revenue.
- Business Subscriptions: Roughly 10% to 15%.
- Individual Subscriptions: About 5%.
Quot 7 minutes ago [-]
The vast majority of my company's enterprise plan use is through Claude Code even though we have access to the API and could be using OpenCode instead.
I don't fully agree with the premise that they intentionally increase system prompts, but the enterprise plan usage is going to make that a huge income for Anthropic.
paxys 7 minutes ago [-]
You're making the opposite argument. Anthropic is incentivized to use less tokens in Claude Code because people are paying a fixed monthly fee for subscriptions.
FuckButtons 6 minutes ago [-]
Nope, that’s not true, because they want you to pay for the higher subscription bracket.
jchook 2 minutes ago [-]
Can confirm — they got me paying $100/mo this way.
Also I think it’s well known that OpenAI is the much less expensive option (in tokens and $$). For the same $20 you get a lot more mileage.
Curious if folks have strong opinions about the overall UX of OpenCode vs CC…
hvb2 11 minutes ago [-]
> This is supported by the fact that they won't let you use your sub on a different coding agent
I mean, that's a very weak argument? Isn't a much more plausible explanation that with your tooling you'll have more of a lock-in than with just your model?
skeledrew 6 minutes ago [-]
I feel like this article isn't saying much. Even with tools disabled, Claude Code still has a crap load of commands and other things that Claude (the model) should know the availability of since it's optimized for them. All of that has to be disabled if this is to be a real harness comparison. And of course the system prompt can be completely replaced, making it a no-brainer to use a more minimal prompt similar to OpenCode. And beyond that nothing else really matters because the rest (cache behavior, etc) lies with the provider's platform, not the harness.
estetlinus 11 minutes ago [-]
Recently switched to Codex after 6m in Claude. Codex seems more open, it’s easier to follow what the model is doing and the approvals have a better UX. Overall, it just feels more transparent. Cost of switching was close to 0.
I don’t like that Claude became more opaque around February, including the system prompts. 33k feels way too much.
jakozaur 36 minutes ago [-]
This isn’t limited to large system prompts. Coding-agent harnesses are also becoming more aggressive about using tools, even for trivial requests. In our tests, prompts such as “Hey” or “commit” sometimes triggered 30+ tool calls:
Tokenflation seems very real: the number of tokens consumed by simple tasks keeps increasing.
prymitive 28 minutes ago [-]
I often find myself annoyed when Opus fixes a typo in a comment and decides to run tests, lints and whenever else it can find to run. Often it will start by stashing current changes just to preemptively check if all tests were passing before.
And I can blame myself a bit because my rules do say: verify all changes with tests. But as there is that I in AI that is hyped which you’d think means it knows not to put tomatoes into fruit salad …
mh- 16 minutes ago [-]
> [..] my rules do say: verify all changes with tests
I am a bit surprised that you're disappointed that it does exactly what you told it to - people usually have the opposite complaint.
If you're using it interactively and watching what it changes, I'd trigger the tests when you think it's needed. And if you want to go more hands-off, why not add try to encode the same nuance you'd use into the rule?
redox99 15 minutes ago [-]
Following rules like "verify all changes with tests" down to a tee is usually a desirable trait in LLMs. Personally I'd leave that behavior there (just like with humans for some tasks like aviation you have them go through checklists even if some stuff you can infer is not needed). But otherwise just make it "always run tests unless you're absolutely sure they can be skipped".
Chris2048 3 minutes ago [-]
> prompts such as “Hey” or “commit” sometimes triggered 30+ tool calls
I read that this is because it wastes time looking through past conversations and other context to figure one what you might want it to do - a less ambiguous prompt would be better.
mft_ 35 minutes ago [-]
Early on in experimenting with local models, I found that hooking them up to Claude Code worked very well, but it was also really slow.
I used mitmproxy (setup assisted by Claude, natch) to capture Claude Code's entire initial system prompt and the whole thing was (I just double-checked) 162k of JSON.
This led me to start experimenting with Pi, OpenCode, and Hermes...
mh- 9 minutes ago [-]
This is interesting, because if I start a fresh session of Claude Code right now and run /context, I see the following:
4k tokens is 15-20kB. I'd ask you to paste that into a gist, but it might have sensitive data in it, because I suspect what you're seeing is not just the system prompt.
alex7o 31 minutes ago [-]
I am forced to use cloude code at work but a good solution is to just use --system-prompt "" and be done with it. I wish they allowed for other harnesses.
AndyNemmity 1 minutes ago [-]
Yep, have been using this for a long time now. No idea why everyone doesn’t.
cube00 8 minutes ago [-]
> --system-prompt ""
Doesn't it need at least a basic system prompt to understand how to call tools?
tyleo 27 minutes ago [-]
I didn’t know you could do this. Is there any analysis of the impact, before and after? I’d love to see some charts of efficacy in real world usage.
alex7o 16 minutes ago [-]
It shows up in /context, but never spend time validating it much. Some people run a proxy to modify their messages.
luciana1u 15 minutes ago [-]
Claude Code sending 33k tokens before reading the prompt is the AI equivalent of a consultant who bills you for the time spent reading your email before they even open it.
estetlinus 14 minutes ago [-]
Well, I have to open the lid on my computer and remember my password, no?
andai 21 minutes ago [-]
With Fable being per token instead of on the subs (unless they changed it again?), I decided to test Claude code on OpenRouter where I had some credits, with Opus 4.8 and Fable 5.
I asked both a trivial question (summarize last commit). Opus cost 50 cents, Fable about $1.
That checks out because Fable's twice as much in the API (though I think its emphasis on correctness makes the difference larger for bigger tasks).
But, at $1 per question, I think I will stick to the subscription for now! I was certainly glad GPT-5.6-Sol is included in OpenAI's subscription, and I'm curious if they'll be able to do the same for GPT-6.
All the VC money appears to have run out a few weeks ago.
andai 14 minutes ago [-]
As for context size and harnesses I did make a trivial bash agent based on this "agent in 50 lines" tutorial[0] recently, and found that for trivial work, it was about an order of magnitude cheaper and faster.
I haven't tested it on anything bigger but it doesn't seem to do the kind of proactive testing, that they do in bigger harnesses.
Codex at least has a system prompt that tells it not to consider a feature a complete until it has verified it. I'm not sure about Claude Code.
I suppose I could add that one line to the prompt, and it would get me much closer to agi :) I think Fable does this proactively even without a prompt, but I haven't tested that yet.
If Fable in my own harness is significantly cheaper than Claude Code, that would be very appealing. (I could actually afford to use it for most things!) But I think most of the cost comes from the testing it does. So we'll have to see.
Fable's subscription inclusion theoretically ends EOD today. Anthropic put a wishy-washy "if we have capacity we'll continue it" thing, and given how competitive GPT 5.6 Sol is, and it is included in OpenAI's subscription, I fully expect Anthropic to extend Fable or they will have a serious exodus on their hands.
Competition is good.
docheinestages 26 minutes ago [-]
I've been trying various harnesses like Pi, OpenCode, Qwen Code, and Nanocoder. A common problem I keep running into is failed tool calls, regardless of the model. What is the best harness and on-device model combination right now?
wolttam 15 minutes ago [-]
> and on-device model combination right now
That would depend entirely on what your device is. This sounds likely not to be an issue with the harness, but the capabilities of the models you've tried.
I experience almost no tool call failure using my nothing-special harness and DSv4 Flash.
arcanemachiner 21 minutes ago [-]
You can't afford the best model. What are your specs and what models + quants have you tried?
Qwen 3.6 35B A3B and Qwen 3.6 27B can both do reliable tool calls on Pi at Q4_K_M using llama.cpp
Maybe related to this minimalism, Pi doesn't come with most of the tools an LLM needs to function efficiently or effectively. I get that a blank slate is the paradigm, and you can add whatever you want, but it's too blank IMO.
arcanemachiner 17 minutes ago [-]
I have a functional Pi config, mostly self-made (it has everything I want, incl. subagents, web search, a /btw command, and other misc. addons), and my system prompt is ~3k.
bel8 14 minutes ago [-]
It's easy to add using plugins.
What do you miss? I ask because I do some heavy work with pi + GLM 5.2 (using opencode Go subscription) and my workflow is plan -> implement.
azuanrb 25 minutes ago [-]
[dead]
ppeetteerr 10 minutes ago [-]
Read through it an I'm curious whether setting the date and cmd on every system prompt call will cause the cache to invalidate.
I guess the cache would only be invalid if the day changed or the root directory, which would technically happen infrequently enough.
anonym29 34 minutes ago [-]
If you really want a minimal agent that you heavily customize, just skip pi (130+ transitive dependencies on the "minimal" pi-coder package) and write your own. You learn a bunch, and it's not hard. You can even ask another LLM to help you get started.
tmalsburg2 7 minutes ago [-]
I wrote my own harness in Emacs and it’s completely ridiculous how well it works. Auto-compact is the only missing feature on my list. Claude‘s approach, if I understand it correctly, invalidates a lot of cached context, and I‘m thinking about a more cache-friendly strategy.
wolttam 14 minutes ago [-]
This is a truly underrated approach IMO
PUSH_AX 16 minutes ago [-]
This is like saying contractor (A) asked for $33,000 to undertake the work and contractor (B) asked for $7,000
Are we measuring and caring about the right thing?
systima 15 minutes ago [-]
Anecdotally, the results from OpenCode + Claude appear to be the same if not better for our uses over the past year.
It works great for long-horizon tasks, and feels like it saves a boatload of tokens.
verdverm 28 minutes ago [-]
The Sleev (the project has been renamed to make a startup) creator was shilling their project in the OpenCode Discord. That person is very convinced they have something that no one has ever built before. They focused on token reduction without any real evals for capability impacts.
I'm generally against this context pruning without prompting or details. Sleev is very opaque about how it works and definitely will bust your cache.
bigyabai 25 minutes ago [-]
It's definitely not unprecedented, but the plugin version is useful. Sleev seems like a nothingburger, I'm happy with the results I get from DCP already.
piokoch 36 minutes ago [-]
No surprise, I've noticed that "agents", not only CC (I am using Copilot) are trying to be "clever", searching for a lot of data. This is good for LLM providers as this eats a lot of tokens.
arcanemachiner 8 minutes ago [-]
OpenAI, to their credit, seems to be focusing pretty heavily on token efficiency in GPT 5.5 and beyond.
MallocVoidstar 37 minutes ago [-]
> Claude Code 2.1.207 and OpenCode 1.17.18, both pinned to claude-sonnet-4-5
So not only is this article AI-written, but the testing was entirely done by AI, too? I can't see any other reason to use such an old model.
> Our traffic passes through a local LLM gateway that wraps requests in its own envelope, a constant we measured at roughly 6,200 tokens with bare calibration requests
Why do you need to do calibration requests to figure out how your own gateway is affecting requests?
> Its subagent lane did not complete cleanly through our gateway
> We attempted to toggle extended thinking in both harnesses and are declining to publish numbers. Our gateway applies its own thinking policy, neither harness's toggle demonstrably survived the path, and anything we quoted would be noise.
Why is your own gateway screwing with your testing?
systima 31 minutes ago [-]
Model:
Cost, mainly. The runs went through a Claude Max subscription rather than metered API billing, and pinning an older stable snapshot kept run-to-run comparisons clean and cheap. The fixed harness payload (system prompt plus tool schemas), so the headline numbers shouldn't change too much.
That said, happy to re-run the matrix on Fable and publish the diff; payload figures should barely move, tool-calling behaviour might.
Gateway:
Meridian (github.com/rynfar/meridian); proxy that bridges the Claude Code SDK to a standard Anthropic endpoint so a Claude Max subscription can drive OpenCode-et-al.
It's the auth route for all agent traffic on the machine, not something built for the benchmark.
We have never read your blog or your content before.
Suspect that many have covered the "Comparing agentic coding tools" angle before, and that the differentiator is depth of analysis + conclusions.
anamexis 19 minutes ago [-]
Do you think you were the first person to write a blog post about coding harness token usage?
slopinthebag 36 minutes ago [-]
Anthropic wants to produce the best coding agent possible and doesn’t care (is even incentivized) about high costs. Other harnesses have to make trade offs between performance and cost.
goda90 32 minutes ago [-]
Given they're incentivized to increase token use, what guarantees that higher token use improves the effectiveness of the agent and isn't just artificial padding?
slopinthebag 24 minutes ago [-]
Well, nothing really. But I assume there can be some benefits to modifying context. For example, updating file contents or marking them as modified, summarization, injecting additional information, removing irrelevant tool call results, etc.
bpye 30 minutes ago [-]
Is there evidence that it is actually a better agent though?
slopinthebag 24 minutes ago [-]
There’s evidence it’s a worse agent actually. I’m just saying in theory.
nubg 18 minutes ago [-]
So? it doesnt matter, after the first turn it's cached. We are probably talking about single digit cents.
Rendered at 19:31:40 GMT+0000 (Coordinated Universal Time) with Vercel.
If I let the main agent do the same task sequentially, it was no problem at all. I don't know if it's really just communication and orchestration that makes sub agents so inefficient, or if Anthropic figured that most people using sub agents pay per token on a big corporate account, so this is an easy way to make more money from tokenmaxxers.
Edit: consumer Claude subs are the 5%. I’d bet most all of CC subs lump in under enterprise.
I don't fully agree with the premise that they intentionally increase system prompts, but the enterprise plan usage is going to make that a huge income for Anthropic.
Also I think it’s well known that OpenAI is the much less expensive option (in tokens and $$). For the same $20 you get a lot more mileage.
Curious if folks have strong opinions about the overall UX of OpenCode vs CC…
I mean, that's a very weak argument? Isn't a much more plausible explanation that with your tooling you'll have more of a lock-in than with just your model?
I don’t like that Claude became more opaque around February, including the system prompts. 33k feels way too much.
https://quesma.com/blog/the-true-cost-of-saying-hi-to-an-ai-...
Tokenflation seems very real: the number of tokens consumed by simple tasks keeps increasing.
I am a bit surprised that you're disappointed that it does exactly what you told it to - people usually have the opposite complaint.
If you're using it interactively and watching what it changes, I'd trigger the tests when you think it's needed. And if you want to go more hands-off, why not add try to encode the same nuance you'd use into the rule?
I read that this is because it wastes time looking through past conversations and other context to figure one what you might want it to do - a less ambiguous prompt would be better.
I used mitmproxy (setup assisted by Claude, natch) to capture Claude Code's entire initial system prompt and the whole thing was (I just double-checked) 162k of JSON.
This led me to start experimenting with Pi, OpenCode, and Hermes...
Doesn't it need at least a basic system prompt to understand how to call tools?
I asked both a trivial question (summarize last commit). Opus cost 50 cents, Fable about $1.
That checks out because Fable's twice as much in the API (though I think its emphasis on correctness makes the difference larger for bigger tasks).
But, at $1 per question, I think I will stick to the subscription for now! I was certainly glad GPT-5.6-Sol is included in OpenAI's subscription, and I'm curious if they'll be able to do the same for GPT-6.
All the VC money appears to have run out a few weeks ago.
I haven't tested it on anything bigger but it doesn't seem to do the kind of proactive testing, that they do in bigger harnesses.
Codex at least has a system prompt that tells it not to consider a feature a complete until it has verified it. I'm not sure about Claude Code.
I suppose I could add that one line to the prompt, and it would get me much closer to agi :) I think Fable does this proactively even without a prompt, but I haven't tested that yet.
If Fable in my own harness is significantly cheaper than Claude Code, that would be very appealing. (I could actually afford to use it for most things!) But I think most of the cost comes from the testing it does. So we'll have to see.
[0] https://minimal-agent.com/
Competition is good.
That would depend entirely on what your device is. This sounds likely not to be an issue with the harness, but the capabilities of the models you've tried.
I experience almost no tool call failure using my nothing-special harness and DSv4 Flash.
Qwen 3.6 35B A3B and Qwen 3.6 27B can both do reliable tool calls on Pi at Q4_K_M using llama.cpp
The entire agent system prompt can be seen here:
https://github.com/earendil-works/pi/blob/main/packages%2Fco...
What do you miss? I ask because I do some heavy work with pi + GLM 5.2 (using opencode Go subscription) and my workflow is plan -> implement.
I guess the cache would only be invalid if the day changed or the root directory, which would technically happen infrequently enough.
Are we measuring and caring about the right thing?
My $20 sub using gpt 5.6 sol thinking-off lasts for hours using pi.
It works great for long-horizon tasks, and feels like it saves a boatload of tokens.
I'm generally against this context pruning without prompting or details. Sleev is very opaque about how it works and definitely will bust your cache.
So not only is this article AI-written, but the testing was entirely done by AI, too? I can't see any other reason to use such an old model.
> Our traffic passes through a local LLM gateway that wraps requests in its own envelope, a constant we measured at roughly 6,200 tokens with bare calibration requests
Why do you need to do calibration requests to figure out how your own gateway is affecting requests?
> Its subagent lane did not complete cleanly through our gateway
> We attempted to toggle extended thinking in both harnesses and are declining to publish numbers. Our gateway applies its own thinking policy, neither harness's toggle demonstrably survived the path, and anything we quoted would be noise.
Why is your own gateway screwing with your testing?
Cost, mainly. The runs went through a Claude Max subscription rather than metered API billing, and pinning an older stable snapshot kept run-to-run comparisons clean and cheap. The fixed harness payload (system prompt plus tool schemas), so the headline numbers shouldn't change too much.
That said, happy to re-run the matrix on Fable and publish the diff; payload figures should barely move, tool-calling behaviour might.
Gateway:
Meridian (github.com/rynfar/meridian); proxy that bridges the Claude Code SDK to a standard Anthropic endpoint so a Claude Max subscription can drive OpenCode-et-al.
It's the auth route for all agent traffic on the machine, not something built for the benchmark.
Suspect that many have covered the "Comparing agentic coding tools" angle before, and that the differentiator is depth of analysis + conclusions.