I quite like the GPT models when chatting with them (in fact, they're probably my favorites), but for agentic work I only had bad experiences with them.
They're incredibly slow (via official API or openrouter), but most of all they seem not to understand the instructions that I give them. I'm sure I'm _holding them wrong_, in the sense that I'm not tailoring my prompt for them, but most other models don't have problem with the exact same prompt.
Does anybody else have a similar experience?
tom1337 14 minutes ago [-]
Yea absolutely. I am using GPT 5.2 / 5.2 Codex with OpenCode and it just doesn't get what I am doing or looses context. Claude on the other side (via GitHub Copilot) has no problem and also discovers the repository on it's own in new sessions while I need to basically spoonfeed GPT. I also agree on the speed. Earlier today I tasked GPT 5.2 Codex with a small refactor of a task in our codebase with reasoning to high and it took 20 minutes to move around 20 files.
furyofantares 4 minutes ago [-]
I don't know any reason to use 5.2, when 5.3 is quite a bit faster.
spiderfarmer 1 minutes ago [-]
If using OpenAI models, use the Codex desktop app, it runs circles around OpenCode.
renewiltord 12 minutes ago [-]
Are you requesting reasoning via param? That was a mistake I was making. However with highest reasoning level I would frequently encounter cyber security violation when using agent that self-modifies.
I prefer Claude models as well or open models for this reason except that Codex subscription gets pretty hefty token space.
nikanj 17 minutes ago [-]
Same, and I can't put my finger on the "why" either. Plus I keep hitting guard rails for the strangest reasons, like telling codex "Add code signing to this build pipeline, use the pipeline at ~/myotherproject as reference" and codex tells me "You should not copy other people's code signing keys, I can't help you with this"
Tiberium 50 minutes ago [-]
I checked the current speed over the API, and so far I'm very impressed. Of course models are usually not as loaded on the release day, but right now:
- Older GPT-5 Mini is about 55-60 tokens/s on API normally, 115-120 t/s when used with service_tier="priority" (2x cost).
- GPT-5.4 Mini averages about 180-190 t/s on API. Priority does nothing for it currently.
- GPT-5.4 Nano is at about 200 t/s.
To put this into perspective, Gemini 3 Flash is about 130 t/s on Gemini API and about 120 t/s on Vertex.
This is raw tokens/s for all models, it doesn't exclude reasoning tokens, but I ran models with none/minimal effort where supported.
And quick price comparisons:
- Claude: Opus 4.6 is $5/$25, Sonnet 4.6 is $3/$15, Haiku 4.5 is $1/$5
- GPT: 5.4 is $2.5/$15 ($5/$22.5 for >200K context), 5.4 Mini is $0.75/$4.5, 5.4 Nano is $0.2/$1.25
- Gemini: 3.1 Pro is $2/$12 ($3/$18 for >200K context), 3 Flash is $0.5/$3, 3.1 Flash Lite is $0.25/$1.5
coder543 14 minutes ago [-]
I wish someone would also thoroughly measure prompt processing speeds across the major providers too. Output speeds are useful too, but more commonly measured.
HugoDias 2 hours ago [-]
According to their benchmarks, GPT 5.4 Nano > GPT-5-mini in most areas, but I'm noticing models are getting more expensive and not actually getting cheaper?
GPT 5 mini: Input $0.25 / Output $2.00
GPT 5 nano: Input: $0.05 / Output $0.40
GPT 5.4 mini: Input $0.75 / Output $4.50
GPT 5.4 nano: Input $0.20 / Output $1.25
karmasimida 21 minutes ago [-]
Those are bigger models. The serving isn’t going to be cheaper.
Why expect cheaper then? The performance is also better
simianwords 2 hours ago [-]
models are getting costlier but by performance getting cheaper. perhaps they don't see a point supporting really low performance models?
HugoDias 1 hours ago [-]
I would be curious to know if from the enterprise / API consumption perspective, these low-performance models aren't the most used ones. At least it matches our current scenario when it comes to tokens in / tokens out. I'd totally buy the price increase if these are becoming more efficient though, consuming less tokens.
BoumTAC 1 hours ago [-]
To me, mini releases matter much more and better reflect the real progress than SOTA models.
The frontier models have become so good that it's getting almost impossible to notice meaningful differences between them.
Meanwhile, when a smaller / less powerful model releases a new version, the jump in quality is often massive, to the point where we can now use them 100% of the time in many cases.
And since they're also getting dramatically cheaper, it's becoming increasingly compelling to actually run these models in real-life applications.
brikym 59 minutes ago [-]
If you're doing something common then maybe there are no differences with SOTA. But I've noticed a few. GPT 5.4 isn't as good at UI work in svelte. Gemini tends to go off and implement stuff even if I prompt it to discuss but it's pretty good at UI code. Claude tends to find out less about my code base than GPT and it abuses the any type in typescript.
patates 4 minutes ago [-]
Big part of these differences may be the system prompts and/or the harness.
pzo 1 hours ago [-]
they do are cheaper than SOTA but not getting dramatically cheaper but actually the opposite - GPT 5.4 mini is around ~3x more expensive than GPT 5.0 mini.
Similarly gemini 3.1 flash lite got more expensive than gemini 2.5 flash lite.
BoumTAC 1 hours ago [-]
But they are getting dramatically better.
What's the point of a crazy cheap model if it's shit ?
I code most of the time with haiku 4.5 because it's so good. It's cheaper for me than buying a 23€ subscription from Anthropic.
philipkglass 48 minutes ago [-]
The crazy cheap models may be adequate for a task, and low cost matters with volume. I need to label millions of images to determine if they're sexually suggestive (this includes but is not limited to nudity). The Gemini 2.0 Flash Lite model is inexpensive and good at this. Gemini 2.5 Flash Lite is also good at this, but not noticeably better, and it costs more. When 2.0 gets retired this June my costs are going up.
cbg0 1 hours ago [-]
Based on the SWE-Bench it seems like 5.4 mini high is ~= GPT 5.4 low in terms of accuracy and price but the latency for mini is considerably higher at 254 seconds vs 171 seconds for GPT5.4. Probably a good option to run at lower effort levels to keep costs down for simpler tasks. Long context performance is also not great.
ryao 2 hours ago [-]
I will be impressed when they release the weights for these and older models as open source. Until then, this is not that interesting.
dack 10 minutes ago [-]
i want 5.4 nano to decide whether my prompt needs 5.4 xhigh and route to it automatically
bananamogul 34 minutes ago [-]
They could call them
something like “sonnet” and “haiki” maybe.
6thbit 47 minutes ago [-]
Looking at the long context benchmark results for these, sounds like they are best fit for also mini-sized context windows.
Is there any harness with an easy way to pick a model for a subagent based on the required context size the subagent may need?
beklein 1 hours ago [-]
As a big Codex user, with many smaller requests, this one is the highlight: "In Codex, GPT‑5.4 mini is available across the Codex app, CLI, IDE extension and web. It uses only 30% of the GPT‑5.4 quota, letting developers quickly handle simpler coding tasks in Codex for about one-third the cost." + Subagents support will be huge.
hyperbovine 54 minutes ago [-]
Having to invoke `/model` according to my perceived complexity of the request is a bit of a deal breaker though.
serf 49 minutes ago [-]
you use profiles for that [0], or in the case of a more capable tool (like opencode) they're more confusing referred to as 'agents'[1] , which may or may not coordinate subagents..
So, in opencode you'd make a "PR Meister" and "King of Git Commits" agent that was forced to use 5.4mini or whatever, and whenever it fell down to using that agent it'd do so through the preferred model.
For example, I use the spark models to orchestrate abunch of sub-agents that may or may not use larger models, thus I get sub-agents and concurrency spun up very fast in places where domain depth matter less.
why isn't nano available in codex? could be used for ingesting huge amount of logs and other such things
machinecontrol 2 hours ago [-]
What's the practical advantage of using a mini or nano model versus the standard GPT model?
aavci 2 hours ago [-]
Cheaper. Every month or so I visit the models used and check whether they can be replaced by the cheapest and smallest model possible for the same task. Some people do fine tuning to achieve this too.
powera 2 hours ago [-]
I've been waiting for this update.
For many "simple" LLM tasks, GPT-5-mini was sufficient 99% of the time. Hopefully these models will do even more and closer to 100% accuracy.
The prices are up 2-4x compared to GPT-5-mini and nano. Were those models just loss leaders, or are these substantially larger/better?
HugoDias 1 hours ago [-]
For us, it was also pretty good, but the performance decreased recently, that forced us to migrate to haiku-4.5. More expensive but much more reliable (when anthropic up, of course).
throwaway911282 1 hours ago [-]
they dont change the model weights (no frontier lab does). if you have evals and all prompts, tool calls the same, I'm curious how you are saying performance decreased..
reconnecting 34 minutes ago [-]
All three ChatGPT models (Instant, Thinking, and Pro) have a new knowledge cutoff of August 2025.
Seriously?
8 minutes ago [-]
dpoloncsak 17 minutes ago [-]
Do you find the results vary based on whether it uses RAG to hit the internet vs the data being in the weights itself? I'm not sure I've really noticed a difference, but I don't often prompt about current events or anything.
reconnecting 6 minutes ago [-]
I noticed that many recent technologies are not familiar to LLMs because of the knowledge cutoff, and thus might not appear in recommendations even if they better match the request.
zild3d 23 minutes ago [-]
whats surprising about that? most of the minor version updates from all the labs are post training updates / not changing knowledge cutoff
reconnecting 8 minutes ago [-]
Thanks for letting me know, I will be waiting for the major update.
F7F7F7 6 minutes ago [-]
It's been like this since GPT 3.5. This is not a limitation and is generally considered a natural outcome of the process.
So there's no major update in the sense that you might be thinking. Most of the time there's not even an announcement when/if training cut offs are updated. It's just another byline.
A 6 month lag seems to be the standard across the frontier models.
reconnecting 37 seconds ago [-]
I've actually started worrying that the amount of false data generated by LLMs on the public internet might provoke a situation where the knowledge cutoff becomes permanently (and silently) fixed. Like we can't trust data after 2025 because so much of it is the result of data poisoning at scale.
casey2 57 minutes ago [-]
I googled all the testimonial names and they are all linked-in mouthpieces.
miltonlost 1 hours ago [-]
Does it still help drive people to psychosis and murder and suicide? Where's the benchmark for that?
system2 1 hours ago [-]
I am feeling the version fatigue. I cannot deal with their incremental bs versions.
Rendered at 19:07:01 GMT+0000 (Coordinated Universal Time) with Vercel.
They're incredibly slow (via official API or openrouter), but most of all they seem not to understand the instructions that I give them. I'm sure I'm _holding them wrong_, in the sense that I'm not tailoring my prompt for them, but most other models don't have problem with the exact same prompt.
Does anybody else have a similar experience?
I prefer Claude models as well or open models for this reason except that Codex subscription gets pretty hefty token space.
- Older GPT-5 Mini is about 55-60 tokens/s on API normally, 115-120 t/s when used with service_tier="priority" (2x cost).
- GPT-5.4 Mini averages about 180-190 t/s on API. Priority does nothing for it currently.
- GPT-5.4 Nano is at about 200 t/s.
To put this into perspective, Gemini 3 Flash is about 130 t/s on Gemini API and about 120 t/s on Vertex.
This is raw tokens/s for all models, it doesn't exclude reasoning tokens, but I ran models with none/minimal effort where supported.
And quick price comparisons:
- Claude: Opus 4.6 is $5/$25, Sonnet 4.6 is $3/$15, Haiku 4.5 is $1/$5
- GPT: 5.4 is $2.5/$15 ($5/$22.5 for >200K context), 5.4 Mini is $0.75/$4.5, 5.4 Nano is $0.2/$1.25
- Gemini: 3.1 Pro is $2/$12 ($3/$18 for >200K context), 3 Flash is $0.5/$3, 3.1 Flash Lite is $0.25/$1.5
GPT 5 mini: Input $0.25 / Output $2.00
GPT 5 nano: Input: $0.05 / Output $0.40
GPT 5.4 mini: Input $0.75 / Output $4.50
GPT 5.4 nano: Input $0.20 / Output $1.25
Why expect cheaper then? The performance is also better
The frontier models have become so good that it's getting almost impossible to notice meaningful differences between them.
Meanwhile, when a smaller / less powerful model releases a new version, the jump in quality is often massive, to the point where we can now use them 100% of the time in many cases.
And since they're also getting dramatically cheaper, it's becoming increasingly compelling to actually run these models in real-life applications.
Similarly gemini 3.1 flash lite got more expensive than gemini 2.5 flash lite.
What's the point of a crazy cheap model if it's shit ?
I code most of the time with haiku 4.5 because it's so good. It's cheaper for me than buying a 23€ subscription from Anthropic.
Is there any harness with an easy way to pick a model for a subagent based on the required context size the subagent may need?
So, in opencode you'd make a "PR Meister" and "King of Git Commits" agent that was forced to use 5.4mini or whatever, and whenever it fell down to using that agent it'd do so through the preferred model.
For example, I use the spark models to orchestrate abunch of sub-agents that may or may not use larger models, thus I get sub-agents and concurrency spun up very fast in places where domain depth matter less.
[0]: https://developers.openai.com/codex/config-advanced#profiles [1]: https://opencode.ai/docs/agents/
Direct image: https://pbs.twimg.com/media/HDoN4PhasAAinj_?format=png&name=...
For many "simple" LLM tasks, GPT-5-mini was sufficient 99% of the time. Hopefully these models will do even more and closer to 100% accuracy.
The prices are up 2-4x compared to GPT-5-mini and nano. Were those models just loss leaders, or are these substantially larger/better?
Seriously?
So there's no major update in the sense that you might be thinking. Most of the time there's not even an announcement when/if training cut offs are updated. It's just another byline.
A 6 month lag seems to be the standard across the frontier models.