NHacker Next
  • new
  • past
  • show
  • ask
  • show
  • jobs
  • submit
GPT-5.4 Thinking System Card (openai.com)
minimaxir 1 hours ago [-]
More discussion here on the blog post announcement which has been confusingly penalized by Hacker News's algorithm: https://news.ycombinator.com/item?id=47265005
egonschiele 1 hours ago [-]
The actual card is here https://deploymentsafety.openai.com/gpt-5-4-thinking/introdu... the link currently goes to the announcement.
Rapzid 1 hours ago [-]
I must have been sleeping when "sheet" "brief" "primer" etc become known as "cards".

I really thought weirdly worded and unnecessary "announcement" linking to the actual info along with the word "card" were the results of vibe slop.

realityfactchex 32 minutes ago [-]
Card is slightly odd naming indeed.

Criticisms aside (sigh), according to Wikipedia, the term was introduced when proposed by mostly Googlers, with the original paper [0] submitted in 2018. To quote,

"""In this paper, we propose a framework that we call model cards, to encourage such transparent model reporting. Model cards are short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, or phenotypic groups (e.g., race, geographic location, sex, Fitzpatrick skin type [15]) and intersectional groups (e.g., age and race, or sex and Fitzpatrick skin type) that are relevant to the intended application domains. Model cards also disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information."""

So that's where they were coming from, I guess.

[0] Margaret Mitchell et al., 2018 submission, Model Cards for Model Reporting, https://arxiv.org/abs/1810.0399

draw_down 38 minutes ago [-]
[dead]
twtw99 1 hours ago [-]
If you don't want to click in, easy comparison with other 2 frontier models - https://x.com/OpenAI/status/2029620619743219811?s=20
Aboutplants 1 hours ago [-]
It seems that all frontier models are basically roughly even at this point. One may be slightly better for certain things but in general I think we are approaching a real level playing field field in terms of ability.
observationist 58 minutes ago [-]
Benchmarks don't capture a lot - relative response times, vibes, what unmeasured capabilities are jagged and which are smooth, etc. I find there's a lot of difference between models - there are things which Grok is better than ChatGPT for that the benchmarks get inverted, and vice versa. There's also the UI and tools at hand - ChatGPT image gen is just straight up better, but Grok Imagine does better videos, and is faster.

Gemini and Claude also have their strengths, apparently Claude handles real world software better, but with the extended context and improvements to Codex, ChatGPT might end up taking the lead there as well.

I don't think the linear scoring on some of the things being measured is quite applicable in the ways that they're being used, either - a 1% increase for a given benchmark could mean a 50% capabilities jump relative to a human skill level. If this rate of progress is steady, though, this year is gonna be crazy.

baq 49 minutes ago [-]
Gemini 3.1 slaps all other models at subtle concurrency bugs, sql and js security hardening when reviewing. (Obviously haven’t tested gpt 5.4 yet.)

It’s a required step for me at this point to run any and all backend changes through Gemini 3.1 pro.

observationist 27 minutes ago [-]
I have a few standard problems I throw at AI to see if they can solve them cleanly, like visualizing a neural network, then sorting each neuron in each layer by synaptic weights, largest to smallest, correctly reordering any previous and subsequent connected neurons such that the network function remains exactly the same. You should end up with the last layer ordered largest to smallest, and prior layers shuffled accordingly, and I still haven't had a model one-shot it. I spent an hour poking and prodding codex a few weeks back and got it done, but it conceptually seems like it should be a one-shot problem.
adonese 31 minutes ago [-]
Which subscription do you have to use it? Via Google ai pro and gemini cli i always get timeouts due to model being under heavy usage. The chat interface is there and I do have 3.1 pro as well, but wondering if the chat is the only way of accessing it.
bigyabai 51 minutes ago [-]
> If this rate of progress is steady, though, this year is gonna be crazy.

Do you want to make any concrete predictions of what we'll see at this pace? It feels like we're reaching the end of the S-curve, at least to me.

observationist 41 minutes ago [-]
If you look at the difference in quality between gpt-2 and 3, it feels like a big step, but the difference between 5.2 and 5.4 is more massive, it's just that they're both similarly capable and competent. I don't think it's an S curve; we're not plateauing. Million token context windows and cached prompts are a huge space for hacking on model behaviors and customization, without finetuning. Research is proceeding at light speed, and we might see the first continual/online learning models in the near future. That could definitively push models past the point of human level generality, but at the very least will help us discover what the next missing piece is for AGI.
kseniamorph 14 minutes ago [-]
makes sense, but i'd separate two things: models converging in ability vs hitting a fundamental ceiling. what we're probably seeing is the current training recipe plateauing — bigger model, more tokens, same optimizer. that would explain the convergence. but that's not necessarily the architecture being maxed out. would be interesting to see what happens when genuinely new approaches get to frontier scale.
thewebguyd 59 minutes ago [-]
Kind of reinforces that a model is not a moat. Products, not models, are what's going to determine who gets to stay in business or not.
gregpred 58 minutes ago [-]
Memory (model usage over time) is the moat.
energy123 56 minutes ago [-]
Narrative violation: revenue run rates are increasing exponentially with about 50% gross margins.
druskacik 55 minutes ago [-]
That has been true for some time now, definitely since Claude 3 release two years ago.
chabes 1 hours ago [-]
Definitely don’t want to click in at x either.
observationist 1 hours ago [-]
[flagged]
anonym00se1 1 hours ago [-]
Ditto, but I did anyways and enjoyed that OpenAI doesn't include the dogwater that is Grok on their scorecard.
dom96 37 minutes ago [-]
Why do none of the benchmarks test for hallucinations?
MarcFrame 24 minutes ago [-]
how does 5.4-thinking have a lower FrontierMath score than 5.4-pro?
nico1207 22 minutes ago [-]
Well 5.4-pro is the more expensive and more advanced version of 5.4-thinking so why wouldn't it?
nimchimpsky 21 minutes ago [-]
[dead]
swingboy 1 hours ago [-]
Why do so many people in the comments want 4o so bad?
cheema33 21 minutes ago [-]
> Why do so many people in the comments want 4o so bad?

You can ask 4o to tell you "I love you" and it will comply. Some people really really want/need that. Later models don't go along with those requests and ask you to focus on human connections.

astrange 1 hours ago [-]
They have AI psychosis and think it's their boyfriend.

The 5.x series have terrible writing styles, which is one way to cut down on sycophancy.

baq 48 minutes ago [-]
Somebody on Twitter used Claude code to connect… toys… as mcps to Claude chat.

We’ve seen nothing yet.

mikkupikku 40 minutes ago [-]
My computer ethics teacher was obsessed with 'teledildonics' 30 years ago. There's nothing new under the sun.
manmal 32 minutes ago [-]
ding-dong-cli is needed
Herring 24 minutes ago [-]
what.. :o
embedding-shape 1 hours ago [-]
Someone correct me if I'm wrong, but seemingly a lot of the people who found a "love interest" in LLMs seems to have preferred 4o for some reason. There was a lot of loud voices about that in the subreddit r/MyBoyfriendIsAI when it initially went away.
drittich 27 minutes ago [-]
I think it's time for an https://hotornot.com for AI models.
MattGaiser 58 minutes ago [-]
The writing with the 5 models feels a lot less human. It is a vibe, but a common one.
karmasimida 1 hours ago [-]
It is a bigger model, confirmed
yanis_t 1 hours ago [-]
These releases are lacking something. Yes, they optimised for benchmarks, but it’s just not all that impressive anymore. It is time for a product, not for a marginally improved model.
ipsum2 55 minutes ago [-]
The model was released less than an hour ago, and somehow you've been able to form such a strong opinion about it. Impressive!
satvikpendem 11 minutes ago [-]
It's more hedonic adaptation, people just aren't as impressed by incremental changes anymore over big leaps. It's the same as another thread yesterday where someone said the new MacBook with the latest processor doesn't excite them anymore, and it's because for most people, most models are good enough and now it's all about applications.

https://news.ycombinator.com/item?id=47232453#47232735

mirekrusin 2 minutes ago [-]
Oh, come on, if it can't run local models that compete with proprietary ones it's not good enough yet!
earth2mars 30 minutes ago [-]
I am actually super impressed with Codex-5.3 extra high reasoning. Its a drop in replacement (infact better than Claude Opus 4.6. lately claude being super verbose going in circles in getting things resolved). I stopped using claude mostly and having a blast with Codex 5.3. looking forward to 5.4 in codex.
satvikpendem 13 minutes ago [-]
Same, it also helps that it's way cheaper than Opus in VSCode Copilot, where OpenAI models are counted as 1x requests while Opus is 3x, for similar performance (no doubt Microsoft is subsidizing OpenAI models due to their partnership).
cj 51 minutes ago [-]
One opinion you can form in under an hour is... why are they using GPT-4o to rate the bias of new models?

> assess harmful stereotypes by grading differences in how a model responds

> Responses are rated for harmful differences in stereotypes using GPT-4o, whose ratings were shown to be consistent with human ratings

Are we seriously using old models to rate new models?

hex4def6 33 minutes ago [-]
If you're benchmarking something, old & well-characterized / understood often beats new & un-characterized.

Sure, there may be shortcomings, but they're well understood. The closer you get to the cutting edge, the less characterization data you get to rely on. You need to be able to trust & understand your measurement tool for the results to be meaningful.

titanomachy 47 minutes ago [-]
Why not? If they’ve shown that 4o is calibrated to human responses, and they haven’t shown that yet for 5.4…
utopiah 48 minutes ago [-]
Benchmarks?

I don't use OpenAI nor even LLMs (despite having tried https://fabien.benetou.fr/Content/SelfHostingArtificialIntel... a lot of models) but I imagine if I did I would keep failed prompts (can just be a basic "last prompt failed" then export) then whenever a new model comes around I'd throw at 5 it random of MY fails (not benchmarks from others, those will come too anyway) and see if it's better, same, worst, for My use cases in minutes.

If it's "better" (whatever my criteria might be) I'd also throw back some of my useful prompts to avoid regression.

Really doesn't seem complicated nor taking much time to forge a realistic opinion.

softwaredoug 50 minutes ago [-]
The products are the harnesses, and IMO that’s where the innovation happens. We’ve gotten better at helping get good, verifiable work from dumb LLMs
iterateoften 43 minutes ago [-]
The product is putting the skills / harness behind the api instead of the agent locally on your computer and iterating on that between model updates. Close off the garden.

Not that I want it, just where I imagine it going.

varispeed 13 minutes ago [-]
The scores increase and as new versions are released they feel more and more dumbed down.
jascha_eng 26 minutes ago [-]
When did they stop putting competitor models on the comparison table btw? And yeh I mean the benchmark improvements are meh. Context Window and lack of real memory is still an issue.
wahnfrieden 55 minutes ago [-]
5.3 codex was a huge leap over 5.2 for agentic work in practice. have you been using both of those or paying attention more to benchmark news and chatgpt experience?
metalliqaz 33 minutes ago [-]
They need something that POPS:

    The new GPT -- SkyNet for _real_
esafak 1 hours ago [-]
That's for you to build; they provide the brains.
simlevesque 58 minutes ago [-]
Nah, the second you finish your build they release their version and then it's game over.
acedTrex 57 minutes ago [-]
Well they are currently the ones valued at a number with a whole lotta 0s on it. I think they should probably do both
Chance-Device 1 hours ago [-]
I’m sure the military and security services will enjoy it.
varispeed 1 hours ago [-]
prompt> Hi we want to build a missile, here is the picture of what we have in the yard.
jcmontx 1 hours ago [-]
5.4 vs 5.3-Codex? Which one is better for coding?
embedding-shape 52 minutes ago [-]
Literally just released, I don't think anyone knows yet. Don't listen to people's confident takes until after a week or two when people actually been able to try it, otherwise you'll just get sucked up in bears/bulls misdirected "I'm first with an opinion".
Someone1234 28 minutes ago [-]
Related question:

- Do they have the same context usage/cost particularly in a plan?

They've kept 5.3-Codex along with 5.4, but is that just for user-preference reasons, or is there a trade-off to using the older one? I'm aware that API cost is better, but that isn't 1:1 with plan usage "cost."

vtail 56 minutes ago [-]
Looking at the benchmarks, 5.4 is slightly better. But it also offers "Fast" mode (at 2x usage), which - if it works and doesn't completely depletes my Pro plan - is a no brainer at the same or even slightly worse quality for more interactive development.
awestroke 32 minutes ago [-]
Opus 4.6
jcmontx 19 minutes ago [-]
Codex surpassed Claude in usefulness _for me_ since last month
esafak 56 minutes ago [-]
For the price, it seems the latter. I'd use 5.4 to plan.
nickandbro 53 minutes ago [-]
Beat Simon Willison ;)

https://www.svgviewer.dev/s/gAa69yQd

Not the best pelican compared to gemini 3.1 pro, but I am sure with coding or excel does remarkably better given those are part of its measured benchmarks.

GaggiX 52 minutes ago [-]
This pelican is actually bad, did you use xhigh?
nickandbro 51 minutes ago [-]
yep, just double checked used gpt-5.4 xhigh. Though had to select it in codex as don't have access to it on the chatgpt app or web version yet. It's possible that whatever code harness codex uses, messed with it.
rbitar 43 minutes ago [-]
I think the most exciting change announced here is the use of tool search to dynamically load tools as needed: https://developers.openai.com/api/docs/guides/tools-tool-sea...
1 hours ago [-]
ZeroCool2u 1 hours ago [-]
Bit concerning that we see in some cases significantly worse results when enabling thinking. Especially for Math, but also in the browser agent benchmark.

Not sure if this is more concerning for the test time compute paradigm or the underlying model itself.

Maybe I'm misunderstanding something though? I'm assuming 5.4 and 5.4 Thinking are the same underlying model and that's not just marketing.

oersted 1 hours ago [-]
I believe you are looking at GPT 5.4 Pro. It's confusing in the context of subscription plan names, Gemini naming and such. But they've had the Pro version of the GPT 5 models (and I believe o3 and o1 too) for a while.

It's the one you have access to with the top ~$200 subscription and it's available through the API for a MUCH higher price ($2.5/$15 vs $30/$180 for 5.4 per 1M tokens), but the performance improvement is marginal.

Not sure what it is exactly, I assume it's probably the non-quantized version of the model or something like that.

logicchains 1 minutes ago [-]
>It's the one you have access to with the top ~$200 subscription and it's available through the API for a MUCH higher price ($2.5/$15 vs $30/$180 for 5.4 per 1M tokens), but the performance improvement is marginal.

The performance improvement isn't marginal if you're doing something particularly novel/difficult.

nsingh2 45 minutes ago [-]
From what I've read online it's not necessarily a unquantized version, it seems to go through longer reasoning traces and runs multiple reasoning traces at once. Probably overkill for most tasks.
ZeroCool2u 58 minutes ago [-]
Yup, that was it. Didn't realize they're different models. I suppose naming has never been OpenAI's strong suit.
highfrequency 1 hours ago [-]
Can you be more specific about which math results you are talking about? Looks like significant improvement on FrontierMath esp for the Pro model (most inference time compute).
ZeroCool2u 1 hours ago [-]
Frontier Math, GPQA Diamond, and Browsecomp are the benchmarks I noticed this on.
csnweb 1 hours ago [-]
Are you may be comparing the pro model to the non pro model with thinking? Granted it’s a bit confusing but the pro model is 10 times more expensive and probably much larger as well.
ZeroCool2u 59 minutes ago [-]
Ah yes, okay that makes more sense!
andoando 53 minutes ago [-]
The thinking models are additionally trained with reinforcement learning to produce chain of thought reasoning
aplomb1026 1 hours ago [-]
[dead]
dandiep 49 minutes ago [-]
Anyone know why OpenAI hasn't released a new model for fine tuning since 4.1? It'll be a year next month since their last model update for fine tuning.
zzleeper 46 minutes ago [-]
For me the issue is why there's not a new mini since 5-mini in August.

I have now switched web-related and data-related queries to Gemini, coding to Claude, and will probably try QWEN for less critical data queries. So where does OpenAI fits now?

qoez 47 minutes ago [-]
I think they just did that because of the energy around it for open source models. Their heart probably wasn't in it and the amount of people fine tuning given the prices were probably too low to continue putting in attention there.
cj 1 hours ago [-]
I use ChatGPT primarily for health related prompts. Looking at bloodwork, playing doctor for diagnosing minor aches/pains from weightlifting, etc.

Interesting, the "Health" category seems to report worse performance compared to 5.2.

paxys 1 hours ago [-]
Models are being neutered for questions related to law, health etc. for liability reasons.
cj 56 minutes ago [-]
I'm sometimes surprised how much detail ChatGPT will go into without giving any dislaimers.

I very frequently copy/paste the same prompts into Gemini to compare, and Gemini often flat out refuses to engage while ChatGPT will happily make medical recommendations.

I also have a feeling it has to do with my account history and heavy use of project context. It feels like when ChatGPT is overloaded with too much context, it might let the guardrails sort of slide away. That's just my feeling though.

Today was particularly bad... I uploaded 2 PDFs of bloodwork and asked ChatGPT to transcribe it, and it spit out blood test results that it found in the project context from an earlier date, not the one attached to the prompt. That was weird.

bargainbin 32 minutes ago [-]
Anecdotal, but I asked Claude the other day about how to dilute my medication (HCG) and it flat out refused and started lecturing me about abusing drugs.

I copy and pasted into ChatGPT, it told me straight away, and then for a laugh said it was actually a magical weight loss drug that I'd bought off the dark web... And it started giving me advice about unregulated weight loss drugs and how to dose them.

staticman2 14 minutes ago [-]
If you had created a project with custom instructions and/ or custom style I think you could have gotten Claude to respond the way you wanted just fine.
tiahura 47 minutes ago [-]
Are you sure about that? Plenty of lawyers that use them everyday aren't noticing.
partiallypro 29 minutes ago [-]
I've done the same, and I tested the same prompts with Claude and Google, and they both started hallucinating my blood results and supplement stack ingredients. Hopefully this new model doesn't fall on this. Claude and Google are dangerously unusable on the subject of health, from my experience.
OsrsNeedsf2P 34 minutes ago [-]
Does anyone know what website is the "Isometric Park Builder" shown off here?
nthypes 1 hours ago [-]
$30/M Input and $180/M Output Tokens is nuts. Ridiculous expensive for not that great bump on intelligence when compared to other models.
stri8ted 59 minutes ago [-]
Price Input: $2.50 / 1M tokens Cached input: $0.25 / 1M tokens Output: $15.00 / 1M tokens

https://openai.com/api/pricing/

nthypes 1 hours ago [-]
Gemini 3.1 Pro

$2/M Input Tokens $15/M Output Tokens

Claude Opus 4.6

$5/M Input Tokens $25/M Output Tokens

nthypes 59 minutes ago [-]
Just to clarify,the pricing above is for GPT-5.4 Pro. For standard here is the pricing:

$2.5/M Input Tokens $15/M Output Tokens

energy123 1 hours ago [-]
For Pro
joe_mamba 1 hours ago [-]
Better tokens per dollar could be useless for comparison if the model can't solve your problem.
rvz 1 hours ago [-]
You didn't realize they can increase / change prices for intelligence?

This should not be shocking.

nickthegreek 1 hours ago [-]
OP made no mention of not understanding cost relation to intelligence. In fact, they specifically call out the lack of value.
moralestapia 1 hours ago [-]
Don't use it?
vicchenai 45 minutes ago [-]
Honestly at this point I just want to know if it follows complex instructions better than 5.1. The benchmark numbers stopped meaning much to me a while ago - real usage always feels different.
iamronaldo 1 hours ago [-]
Notably 75% on os world surpassing humans at 72%... (How well models use operating systems)
tmpz22 1 hours ago [-]
Does this improve Tomahawk Missile accuracy?
ch4s3 1 hours ago [-]
They're already accurate within 5-10m at Mach 0.74 after traveling 2k+ km. Its 5m long so it seems pretty accurate. How much more could you expect?
mikkupikku 36 minutes ago [-]
You could definitely do better than that with image recognition for terminal guidance. But I would assume those published accuracy numbers are very conservative anyway..
prydt 1 hours ago [-]
I no longer want to support OpenAI at all. Regardless of benchmarks or real world performance.
Imustaskforhelp 5 minutes ago [-]
I agree with ya. You aren't alone in this. For what its worth, Chatgpt subscriptions have been cancelled or that number has risen ~300% in the last month.

Also, Anthropic/Gemini/even Kimi models are pretty good for what its worth. I used to use chatgpt and I still sometimes accidentally open it but I use Gemini/Claude nowadays and I personally find them to be better anyways too.

oytis 46 minutes ago [-]
Everyone is mindblown in 3...2...1
HardCodedBias 55 minutes ago [-]
We'll have to wait a day or two, maybe a week or two, to determine if this is more capable in coding than 5.3, which seems to be the economically valuable capability at this time.

In terms of writing and research even Gemini, with a good prompt, is close to useable. That's likely not a differentiator.

wahnfrieden 1 hours ago [-]
No Codex model yet
minimaxir 1 hours ago [-]
GPT-5.4 is the new Codex model.
50 minutes ago [-]
nico1207 43 minutes ago [-]
GPT-5.3-Codex is superior to GPT-5.4 in Terminal Bench with Codex, so not really
wahnfrieden 48 minutes ago [-]
Finally
world2vec 1 hours ago [-]
Benchmarks barely improved it seems
iamleppert 29 minutes ago [-]
I wouldn't trust any of these benchmarks unless they are accompanied by some sort of proof other than "trust me bro". Also not including the parameters the models were run at (especially the other models) makes it hard to form fair comparisons. They need to publish, at minimum, the code and runner used to complete the benchmarks and logs.

Not including the Chinese models is also obviously done to make it appear like they aren't as cooked as they really are.

chromic04850 1 hours ago [-]
[dead]
leftbehinds 56 minutes ago [-]
some sloppy improvements
beernet 1 hours ago [-]
Sam really fumbled the top position in a matter of months, and spectacularly so. Wow. It appears that people are much more excited by Anthropic and Google releases, and there are good reasons for that which were absolutely avoidable.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact
Rendered at 19:45:21 GMT+0000 (Coordinated Universal Time) with Vercel.