Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
joefourier 8 hours ago [-]
> current frontier models
> Gemini 2.5 Pro, O3, Claude Sonnet 3.7 and ChatGPT 4.1
The gap in capabilities between those models which they tested, and actual current frontier ones is enormous. I would not trust that any conclusions they made are applicable.
21asdffdsa12 2 minutes ago [-]
So give me a falsifiable point in time, a model you would claim succeeds at the task. One does not get to hotfix-patch updater out of the pressures of reality.
Today is the day.
sobellian 7 hours ago [-]
I tested both myself and a weak bot against Astra xhigh, https://lichess.org/study/27lCQqDa. It's still pretty bad at chess, though it takes longer to devolve into illegal moves.
aprilthird2021 41 minutes ago [-]
They still need supervision though
sigmoid10 8 hours ago [-]
The actual current frontier plays somewhere around GM level.
It's also worth noting that the very latest models (GPT-6 and Fable 5.1) actually play worse than their immediate predecessors, so it is likely that the labs are not benchmaxxing for this yet. If they did, I'm sure they could come up with something superior to humans. But there is probably very little demand for this compared to IT stuff.
minraws 7 hours ago [-]
I know HN readers and posters just read numbers and can't be bothered to read, but please read the methodology before making any claims.
> About their ELO ratings from their own website:
> A field-relative rating calculated within ChessBench. It compares performance among the tested models and is not a direct equivalent of a human chess rating.
I am around 1600 elo in over the board I can mop up Astra Fable etc even if I give them literal infinite time and all the subagents and internet access..
Please folks at least use your AIs to read stuff before making claims.
AI is not GM level, it's not even 1600, I am 1600 by using memorized openings people frequently fall for with very basic intuitions.
A GM is 2600 they can beat me in under 20 moves...
Why do I even scroll through this website. For a moment I truly felt fooled, but then I read like a human should.
Maybe I should stop doing that will be a happier life, don't think just believe in the AGI.
automatic6131 3 minutes ago [-]
HackerNews is Gell-Mann amnesia that refreshes on every comment on every thread.
Onavo 8 minutes ago [-]
> even if I give them literal infinite time and all the subagents and internet access..
Don't use the word infinite in any CS claims. They can recreate or approximate monte Carlo tree search and it technically is still a correct solution in your framing of the problem so long they defeat you.
uncivilized 7 hours ago [-]
HN is no different than Reddit, or any social media for that matter, in that commenters pretend to read articles.
nalekberov 40 minutes ago [-]
Exactly, especially when you touch “forbidden things”, like questioning why rust IS not the best system programming language, you will be punished so hard by “expert”s.
xdavidliu 6 hours ago [-]
that is if it even a human commenter at all
linkjuice4all 5 hours ago [-]
State-sponsored psyop meta comments aside, the models obviously continue to get better, but there is still a lot of 'guard railing' required to keep even the latest models completely on-task. The chess example is interesting because it's clearly a well-studied and established domain so the rules, strategies, and whatever else is in the training data should make yield excellent results; but clearly there is some behavior in these systems that's difficult to engineer out.
peab 7 hours ago [-]
What levels are they actually at in your experience?
minraws 7 hours ago [-]
Sub 1300 that's my rating in the singular official tournament I participated at.
But given how easily I can crush them and how often they want to make illegal moves (btw above bench seems to use a harness that pokea the model until it gives valid moves).
I would rate them around 500-800 big range but at that level it's all about if the model can recall an opening or not. If it plays good first 4-8 moves the person on the end will fumble for certain and they win.
I can play good/best moves till 14-15 moves if I remember the lines and find someone who falls for it.
If you could give them the lines as prompts like the best 20-30 openings then they will be around 700-800.
700 is around the rating for a human who doesn't know the tricks but can do bare minimum calculations and understands the rules thoroughly.
Forgeties79 7 hours ago [-]
As someone who used to compete for years and plays currently as a hobbyist, you’re absolutely correct. LLM’s are terrible at chess and if anyone wants to sober up their view on AI, try it yourself.
Anyone who casually plays on a regular basis can beat them more often than they lose. As you said if you just know the core openings (and end games, both of which you can get a handle on with modest effort) you will generally win.
Edit: reminder we had computers beating the best players in the world literally decades ago. LLM’s are remarkable tools but the current promises and expectations are ridiculous
zug_zug 6 hours ago [-]
So you can see an actual game on that website, and the play seems pretty decent to me for a while (~1700 lichess = 1300 elo) until move 28 when black throws away their queen for absolutely no reason in an incomprehensible blunder.
In some ways this is reflective of the AI experience at large, sometimes shockingly competent but then also sometimes ludicrously incompetent.
echelon 7 hours ago [-]
The AI can write a chess bot program that will beat you.
You're thinking about this the wrong way. The system is built and delivered as it is because that's how the providers make the most money. If they cared to have it perform well in chess games, you'd see a different shape and behavior.
We shouldn't ask the multibillion dollar automated software generation system to play games with us any more than we should ask a Boeing's flight guidance system to do so.
I believe I remember reading it was based on Glaurung's code (which eventually evolved into what we now know as the juggernaut Stockfish).
what 7 hours ago [-]
I can write a chess bot program that will beat you. Does that mean I’m good at chess?
>If they cared to have it perform well in chess games, you'd see a different shape and behavior.
So the things they claim are on the verge of AGI actually aren’t? They need to be trained for specific tasks?
phoghed 6 hours ago [-]
They’ll never be AGI simply because the definition will be constantly updated to be some steps ahead of them.
jibal 31 minutes ago [-]
First, you're moving the goalposts. Second, it's not actually true that any existing frontier AI can write a chess bot program that can beat a 1600 player ... not unless the program is derived from Stockfish or some other leading engine that has been in development for decades.
> The system is built and delivered as it is because that's how the providers make the most money. If they cared to have it perform well in chess games, you'd see a different shape and behavior.
These comments indicate a complete failure to understand the technology.
I won't respond again.
minraws 7 hours ago [-]
So AGI needs to be trained on something to work well on it. Lovely reasoning we have right here.
Delusion runs deep in HN circles.
I say that as someone heavily invested in AI startups and projects and as someone working in the field.
I think most people on HN should touch grass and find real human contact. Lmao
Incredible reasoning all around here.
diehunde 6 hours ago [-]
AI bros: the LLM beats humans at solving Navier-Stokes and some old cypher. We are close to AGI
Also AI bros: LLM can’t beat an avg chess player. But that doesn’t mean anything. It doesn’t count
hackinthebochs 6 hours ago [-]
>LLM can’t beat an avg chess player.
Why should that matter?
lelanthran 58 minutes ago [-]
> Why should that matter?
Because we want to use this as a replacement for humans, and the average human can learn the rules of chess without needing to see the rules explained hundreds of thousands of times in millions of games.
So, yeah, it matters if a model has millions of examples of something in its training set and still cannot follow the rules.
hackinthebochs 42 minutes ago [-]
We're not talking about learning the rules of chess here, but playing a competent game from just being shown the rules. Why is it so hard for people to keep track of the thread of discussion?
janalsncm 5 hours ago [-]
If something has general intelligence it should be able to read the rules of a game and follow them. Therefore an artificial general intelligence (AGI) should be able to do this.
So we have a situation where very powerful and influential people are saying we will have AGI in 6 months (if we don’t already), yet the facts on the ground are so clearly pointing in the opposite direction.
hackinthebochs 5 hours ago [-]
I would bet a lot of money that Astra can follow the rules of chess (perhaps if repeated within the context window). Also, this is a different argument than what I responded to.
minraws 4 hours ago [-]
I can write you a benchmark to prove it even with a heavy handed system prompt Astra will make an illegal move during the course of the games first few moves are generally ok since it's just throwing out learned moves.
hackinthebochs 43 minutes ago [-]
I'd genuinely like to see the results of that.
janalsncm 3 hours ago [-]
I would definitely take you up on that.
lostmsu 5 hours ago [-]
The fact that LLMs can play chess at any level is a strong indication we are in AGI.
recursive 5 hours ago [-]
Can they if they frequently make illegal moves?
bigstrat2003 4 hours ago [-]
No it isn't. Computers could play chess long before LLMs, better than LLMs can in fact. That didn't make them AGI.
echelon 7 hours ago [-]
I'm stating that certain folks are trying to use the software-generating product as an AGI/ASI and then complaining when it doesn't play chess very well.
People are holding it wrong, deliberately or not. Some are inventing bad faith measures so they can claim AI sucks.
minraws 6 hours ago [-]
Then why respond at all for the sake of responding?
We all know AI can code, but the question it all stemmed from what if it's AGI or GM level in chess on it's own.
You can't just back pedal from the statement that apparently being able to code a chess engine is the same as being good at chess.
I can write a chess engine that beats Magnus Carlson without AI that alone neither makes me GM level or AGI or any of the other claims the above comments seem to be making?
sdf32dsf 6 hours ago [-]
He keeps posting with a particular type of tone.
He definitely needs to touch grass.
echelon 4 hours ago [-]
Try to embrace hacker ethos and stop hating.
Y'all seem to miss the point of this forum. Building and hacking and science and engineering.
I swear there's a whole lot of you who just like to look down instead of up. There's a whole universe up there.
modulus1 6 hours ago [-]
I agree w/ this perspective. An agent with a harness that can run programs can solve a lot more than one without the harness. The AI system includes the harness, and it's not clear to me that AGI requires more than LLMs + code generation & execution are capable of.
minraws 5 hours ago [-]
So AI is AGI in fields where code can't solve anything?
Is code omnipotent, I have been in software all my life and I would hard agree here.
Sure stuff LLMs can do with being good at parts of code reproduction is incredible. And honestly it's the new way to do a lot of things but I have not see an iota of proof that it can scale across the board.
For instance Maths is just code with different symbols and slightly less universally legible concepts.
AI is the best invention at figuring out or walking the search space and directionally doing logically computation over general software adjacent stuff.
But that's it, I am certain a bunch of companies will make a lot of money despite no AGI.
I think people either don't understand AGI or don't understand how real world works.
Until an LLM can bow it's head take responsibility for mistakes made and ensure they aren't repeated again with 100% confidence to the leadership it's inarguably a tool a rather questionable one at that.
csande17 7 hours ago [-]
Even if you take that website at face value, the ELO scores shown are relative to the other AI models tested, and not comparable to the ELO scores of humans who play against other humans.
MichaelNolan 7 hours ago [-]
I wonder why they didn’t throw a real chess engine in there for a baseline. There are engines where you can set the elo in the settings, so it should possible to see these LLMs relative to a human 1500 rather than just relative to each other.
shric 7 hours ago [-]
> so it should possible to see these LLMs relative to a human 1500 rather than just relative to each other
As a 1500 elo human I can tell you that a 1500 elo chess engine doesn't play like anything like a 1500 elo human.
traes 6 hours ago [-]
This is true, but I'm not sure it matters? I was poking around at the lichess database recently and those elo calibrated bots are
remarkably well calibrated, their rating variance sticks out like a sore thumb compared to human players even at similar game volumes. So it should still be a decent predictor of how good a human at that level is, even if the playstyle seems alien.
sashank_1509 7 hours ago [-]
These ratings seems very wrong, i have beaten GPT Astra max thinking in chess and my rating is close to 1500. The ratings here seem more accurate: https://chessbenchllm.onrender.com/
GPT-6 almost never suggests an illegal move anymore while even Sol still did so time to time
jibal 18 minutes ago [-]
"Elo is relative to the ChessBench field."
They are of course "wrong" if you don't read the faint fine print and sensibly interpret them as FIDE or similar ratings.
einszwei 7 hours ago [-]
Probably tells us that without labs explicitly training/tuning the models or designing the harness (with fast oracle) the LLMs aren't going to get good at those areas.
more like you lose intelligence in chess by maxing for coding... hence knocking back the claims of emergent intelligence
jibal 37 minutes ago [-]
Please do not post misinformation. They are not playing anywhere near GM level.
"Elo is relative to the ChessBench field."
8 hours ago [-]
8 hours ago [-]
1dom 12 minutes ago [-]
The last post on HN I read was about someone using LLMs to reverse engineer an Apple GPU driver for linux in a month. The top comment points out how the poster must have had specialist internal domain specific contact with Apple. But then the thread concludes that wasn't the case and that this would take domain experts years to do.
> "current frontier models need laborious oversight and guardrails on even the simplest tasks"
I feel this statement is extreme. I can't personally reconcile it with any of the projects we're regularly seeing get delivered largely by LLMs now.
What are you thoughts? Like, what's your position here? Even if you sincerely believe frontier models need laborious oversight on even the simplest of tasks, do you think that accurately captures and reflects the current state and progress of frontier LLMs?
Don't get me wrong, there's lots of things LLMs can't do well, but the idea that they're basically not helpful for even the simplest of tasks seems... disingenuous?
WhitneyLand 7 hours ago [-]
1. It’s hard to trust a 2026 paper that’s showing results for such old models.
2. Chess seems to be a poor benchmark for generalized strategic reasoning. People who are good at it rely more on experience and deep domain expertise than on skills that generalize to make them experts at unrelated tasks.
3. The study sounds like proving humans will never fly because they don’t have wings. In reality, humans do fly, and Claude Fable would destroy any human at chess by coding a strong enough engine on the fly.
Certhas 25 minutes ago [-]
Good science, properly digested and presented takes time.
The idea that anything other than a breathless blog post about the latest model snapshot is useless is really poisonous to proper debate on AI issues
paimapi 6 hours ago [-]
so prove it! get a public repo out there, have it play against some open source engines
also I think the operative letter in AGI is the G - and if the G is short for 'variably competent savant-like hyperfocus on certain kinds of software coding and not any other general skill' then its not really G at all, is it?
BobbyJo 6 hours ago [-]
I suck at chess. Are you saying I can't be intelligent?
nmehner 1 hours ago [-]
If you read all chess tutorials, strategy documentation and game archives on the internet and then would still suck at chess: yes.
paimapi 5 hours ago [-]
is that what I'm saying? or am I talking about AGI? perhaps there's some irony here to be explored when it comes to basic reading comprehension gaps
jibal 12 minutes ago [-]
That's a polite way to put it. :-)
carodgers 6 hours ago [-]
> Claude Fable would destroy any human at chess by coding a strong enough engine on the fly.
A bash script can clone and build stockfish, feed in human moves, and reply. By your standard, this bash script would "destroy any human at chess."
Are you interested in assessing the intelligence of the model, or the intelligence of the tools the model can use?
nimbleal 15 minutes ago [-]
Maybe practically it doesn’t matter? Perhaps AGI is not the model but the model plus everything it’s got access to. If we’re modelling intelligence in the way we seem to have to to have any coherent definition of AGI, it seems to me <model + everything it can access> is always going to be more “intelligent” than <model> alone.
manquer 7 hours ago [-]
> People who are good at it rely more on experience and deep domain expertise
People are good are 1900 or 2100 above and the top ones who spend decades in the field i.e. deep expertise are well in the 2200-2700 range.
A 1100 player is none of these things, they are purely relying on strategic reasoning there is a good chance they cannot name a single opening or articulate clearly why a move was appropriate. 1100 is quite low bar.
svachalek 5 hours ago [-]
1100 at online speed chess or something, could be. I'm not that deep in the chess world but everyone I know that can make 1100 in official rating can name a dozen openings and most of the known tactics, and is pretty good at applying at least one opening.
what 7 hours ago [-]
> Claude Fable would destroy any human at chess by coding a strong enough engine on the fly.
Delusional, but then Claude fable also isn’t beating any human at chess, the engine is.
gizmodo59 7 hours ago [-]
why cant models make a tool call to stockfish? its like saying model can't execute python for complex math calculations
what 6 hours ago [-]
Because then it’s not playing chess, stockfish is?
redcheeks 6 hours ago [-]
Exactly. All these nerds saying cars make bad submarines. Well duh.
wat10000 8 hours ago [-]
I wonder how current models would fare. The ones they tested are fairly old now.
consensus1 8 hours ago [-]
This isn't how intelligence works. The LLM may not be able to play chess directly through inference, but it can write a program to do it and execute that program. Same as how human intelligence works. We can't fly, but we can build planes.
thesmtsolver2 7 hours ago [-]
Human beings can play chess directly without coding up a tool.
jstanley 6 minutes ago [-]
Asking an LLM to play chess by writing algebraic notation is like asking a human to play chess blindfolded.
Yes some people can do it but most people can't even if they're unusually intelligent.
You really need to be giving the LLM a board representation.
qarl 2 hours ago [-]
If they wanted to train an LLM to play chess they could easily do so.
But nobody wants that.
consensus1 6 hours ago [-]
Very poorly compared to the tools we have built. Similar to the LLM.
thesmtsolver2 5 hours ago [-]
Comparing to raw LLMs? Much much better.
shimman 6 hours ago [-]
Poorly in what sense? I think human chess leagues are way more popular and fun than just playing a computer by yourself. Human oriented communities are always a vastly better experience than their digital counterparts.
There's more to games than simply winning you know.
nefarious_ends 8 hours ago [-]
Thanks for saying this, feels like everyone has gone insane over this stuff.
what 7 hours ago [-]
Humans don’t code a $game engine to play $game, they can just play it. It seems like you are the one that has gone insane.
hackinthebochs 6 hours ago [-]
And how many years of direct play and study does it take for a human to get good at chess or any other game? Absolutely no human ever could be good at chess just by reading a few books, or even every book on chess. That's just not how the brain works. If LLMs could do that they would truly be superintelligence.
sph 1 hours ago [-]
No, learning is definitely not a sign of super intelligence. I know words don’t mean anything anymore, but that is simply general intelligence, despite the claims we have reached this milestone.
hackinthebochs 45 minutes ago [-]
No, but superhuman capabilities derived from ordinary learning is, which is what the parent comment described. Why is that not obvious?
threethirtytwo 8 hours ago [-]
The story isn't so clear cut.
The caveat is: It depends on the task.
Are there reams of chess moves that the model can train off of? No.
Are there reams of math papers the model can train off of? Yes.
vmg12 8 hours ago [-]
> The caveat is: It depends on the task.
I think the line of criticism around LLMs sucking at chess makes more sense when you understand what the AI companies are saying about the future trajectory of these models.
The entire recursive self improvement story falls apart once you point out that there is not much "cross domain transfer learning". Meaning that training an LLM to become good at coding, math, etc, will eventually transfer into them being good at other skills that were not explicitly trained for.
Using games like chess which have little economic value is actually a good test for this. What's even more surprising about them sucking at chess is how much information about chess strategy exists in the training data.
FuckButtons 7 hours ago [-]
There’s multiple databases of games in algebraic notation. You can also, very easily rl train on pitting models against one another, even without mcts.
tjwebbnorfolk 8 hours ago [-]
> Are there reams of chess moves that the model can train off of? No.
This is as false as something can possibly be. There are open databases of millions of chess games spanning hundreds of years.
XenophileJKO 8 hours ago [-]
It is even worse.. This is a classical reinforcement problem where data generation is easy because the rule set is pre-defined. So you really don't even need any data to start with (but would help).
manquer 7 hours ago [-]
There are more possible game combinations than atoms in the universe, even those generation of valid game states are as you say pre-defined. that is why models cannot go this route and therefore are poor at chess
wat10000 5 hours ago [-]
Isn’t this exactly how AlphaZero was trained? The rules are known and well defined so the training process can generate games without any outside data.
The only reason LLMs are this bad at chess is because the labs don’t care about chess performance so they’re not going out of their way to train the models for it. The ability they do have is from what chess information happens to be in the training data, plus whatever general reasoning abilities they may be able to apply.
freejazz 8 hours ago [-]
>Are there reams of chess moves that the model can train off of? No.
For real??
8 hours ago [-]
iwontberude 8 hours ago [-]
[dead]
keephnacct 8 hours ago [-]
[flagged]
8 hours ago [-]
famouswaffles 8 hours ago [-]
Frontier labs don't care about chess. If OpenAI cared, GPT-7 could be a grandmaster+ level chess player. In fact there's a google paper on grandmaster level chess without search with a 270M transformer. Outside that, there was gpt-3.5-turbo instruct which was incidentally a 1800 lichess elo player that didn't make any illegal moves even after a few thousand moves.
Frontier labs care deeply about automating knowledge work and computer use. They are working hard on getting models better and better, and they are succeeding. Astra is a step change on that front. So good luck i guess, if chess performance is your barometer.
bigstrat2003 7 hours ago [-]
> Frontier labs don't care about chess. If OpenAI cared, GPT-7 could be a grandmaster+ level chess player.
If the models were actually intelligent, the way that the boosters claim, they wouldn't need to be tuned to play chess in order to be good at it. That's kind of the point of intelligence, that it is generically applicable to whichever task one wishes.
skydhash 7 hours ago [-]
Pretty much this. Feed it a book or two on chess, and you should have a decent (or good) player. That's the generic intelligence people have. The aims is not to be supremely talented at something, but being able to read a manual and figure how to use/play something. Mastery can be gained overtime.
willmarch 7 hours ago [-]
If you gave a human a book or two on chess they would not become a decent player (they would be closer to 500-600 than 1100 ELO) and they would only get better after playing hundreds or thousands of games (often making illegal moves and moves that violate the rules of chess as they learn).
Your assumptions/intuition about generic human intelligence feels quite incorrect, considering LLMs currently play better than a brand new human player would (presumably without any attempt to fine tune them specific on chess, such as playing thousands of games).
rsfern 5 hours ago [-]
the discussion isn’t really about whether language models can become strong chess players though, the point is they seem to struggle to consistently make valid moves. Most humans don’t need to read two books to pick that up, just a couple lines of basic instructions
willmarch 4 hours ago [-]
That has not been my experience with new players, they regularly make invalid or incorrect moves even after detailed instructions especially in novel situations.
what 6 hours ago [-]
> considering LLMs currently play better than a brand new human player would
They’ve ingested all the literature on playing chess, a brand new human player has not.
willmarch 6 hours ago [-]
Yes, but my point is that humans can’t even do the thing that the above comments are claiming humans can do (read a book or two and be decent at chess), and then they complain that LLMs can’t do the same thing (that humans can’t do either).
We seem to be moving goalposts to the point that humans don’t even live up to the expectations of the AI critics. The only way you get better at chess is by playing a lot of games and learning from mistakes, that goes for humans or AI agents, not simply by reading about chess.
skydhash 5 hours ago [-]
> The only way you get better at chess is by playing a lot of games and learning from mistakes
How can you play without being aware of the rules and how can you learn from your mistakes without knowing they are mistakes? That’s what I said about reading a book of two. It is to kickstart the process. Then mastery is gained over time through practice.
This kickstarting then gradual refinement is how most people learn. And the foundational knowledge stays. Even a basic player knows to not do illegal moves.
willmarch 4 hours ago [-]
Reading can kickstart the process, but you can also make random moves guided by some sort of system (such as a computer GUI) or learn by watching other players play. The overall point is that you learn through observation and lots of trial and error (whether you are a human or a computer). And beginners in chess often make illegal moves even after learning the rules, it's fairly common.
It feels like you're trying to say that humans never make illegal moves while learning chess, which doesn't match with my experience. I'm trying to understand your overall point.
famouswaffles 7 hours ago [-]
If humans were actually intelligent, they wouldn't need to train and practice to play good chess. I mean, what level do you think people without any practice or training are ?
diehunde 6 hours ago [-]
Except all these LLMs were already trained with hundreds of chess book and game databases and they still suck
famouswaffles 6 hours ago [-]
If all you do is read chess books, you'll be a shit player. Training and practice is what it takes to be great.
diehunde 5 hours ago [-]
Oh right. But if all you do is reading programming books you are an amazing programmer? Where is all the training and practice LLMs did to become so good at coding?
hackinthebochs 25 minutes ago [-]
>Where is all the training and practice LLMs did to become so good at coding?
Coding is a matter of translating the natural language description of a problem to the code specification while keeping the semantics fixed (and imputing the unspecified semantics as necessary). It is not considerably more difficult than translating between two dissimilar natural languages. Chess isn't a matter of language translation, but a compute heavy game of finding the best move out of many possibilities with wide variation in the quality of each move. Chess takes directed practice and reinforcement whereas language translation does not.
famouswaffles 5 hours ago [-]
LLMs (and Humans) don't get really good from programming books lol. The training and practice is the actual code they predict and learn from in the process of predicting.
diehunde 4 hours ago [-]
Oh I see. So if someone just reads books AND actual code then they can become experts, got it. And by the way LLMs are also trained with probably hundreds of thousands of actual games not just books
sdf32dsf 6 hours ago [-]
WTF even is this post?
WarmWash 6 hours ago [-]
Contrary to popular belief, you need a lot of training on something for an LLM to be good and consistent with it.
People think that if one mention exists in the training set, then the LLM is perfect at it.
diehunde 5 hours ago [-]
Not one mention. Hundreds of books, articles and databases of games.
bigstrat2003 4 hours ago [-]
OpenAI making the next model good at chess is not analogous to a human training to get good at chess. It is analogous to God creating Human 2.0 which now has increased chess playing ability. If LLMs were intelligent the way humans are, then the models that exist right now would be able to spend time improving themselves at chess and become good at it. They can't do this because they are not, in fact, intelligent.
matteoraso 7 hours ago [-]
I don't see why this is such a big deal. Nobody's using LLMs for chess, but even if they are, just give them Stockfish as part of their harness. They don't need to do everything themselves as long as they're intelligent enough to use tools.
TrackerFF 22 minutes ago [-]
The challenge with estimating abilities, is that we don’t know what the models can achieve if we just burn enough money. The navier-stokes shows us what mathematical problem can be solved when $10m worth of compute is thrown at something.
It also makes one wonder: What could AI solve if we managed to orchestrate billions worth of agents to take on a specific problem?
IMO the very best case scenario / potential for these are likely better than we think, but right now hidden due to logistical and financial reasons.
But if we assume that the model costs will continue to drop by a factor of 5-10 annually, there will always be a latency of a couple of years between what is completely out of reach, and what is financially viable.
Basically: If you knew AI could be affordable enough in 3-5 years so that even the most underfunded researchers could use it to solve cancer, how much would you value it now?
keeda 7 hours ago [-]
The premise in the very first point seems off:
> the frontier labs are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers...
Even assuming this is how the AI companies are being valued (they're not), the numbers are off.
The "value" of most knowledge workers -- based on what enterprises currently pay for them -- is $50 - 70 trillion annually. It's reasonable to assume that if AI drop-in-replaced all those knowledge workers, AI companies could credibly charge somewhere in that order of magnitude, because that's what the market is already bearing.
So if their hypothetical revenues are double-digit trillions and valuations are some multiple of that, the entire AI industry would be valued at double-digit trillions at the least.
Yet cumulatively the industry (the frontier labs + the SWAG estimate of the AI parts of all the other players) are valued at, say, ~6 - 7 trillion? Which seems like a fair approximation of how much knowledge work they can currently automate.
zug_zug 5 hours ago [-]
> The "value" of most knowledge workers -- based on what enterprises currently pay for them -- is $50 - 70 trillion annually.
What do you mean? The sum of ALL US salaries is $13.4 Trillion per year. According to google $65T is the sum of ALL salaries Globally (not just knowledge workers). It's not reasonable to assume AI is a drop-in-replacement for any job yet (perhaps bottom tier customer support from oversees?).
> So if their hypothetical revenues are double-digit trillions and valuations are some multiple of that
So you're sort of premising here than more than 16% or 1/6 of all the world's jobs get replaced by AI. Hopefully you can understand that's both not the current AI capability and also would be a terrible (unprecedented?) economic shock.
keeda 1 hours ago [-]
It does not have to be 16% of all jobs, but 16% of any given job, i.e. AI stays in an augmentative role rather than a complete job automation. The simplistic analysis is if a tool makes you X% faster, that can be worth X% of your salary to your employer.
Unfortunately, I do fear that AI adoption will go beyond augmentation to automation, and I do fear an economic shock. Just posted this down-thread: https://news.ycombinator.com/item?id=49722616
farrellm23 4 hours ago [-]
You are comparing company valuations to annualized revenue (as approximated by some fraction of total knowledge worker compensation). Valuations are (roughly) based on the sum of all discounted future cash flows, not just the current year’s revenue.
iron_albatross 6 hours ago [-]
When thinking about these valuations, shouldn’t we try to quantify how much knowledge work becomes obsolete if other knowledge workers are automated? I.e. there are a huge amount of knowledge workers employed in businesses that create tools for other knowledge workers. AI won’t automate their work, those businesses will just cease to exist.
And then there’s the second order effect: if all the knowledge workers get automated, who is going to buy the stuff that’s produced?
credit_guy 6 hours ago [-]
I think you are committing the lump of labor fallacy [1]. Lots of jobs will disappear, but others will appear. Lots of things (both intellectual and material) that are produced nowadays by humans will be produced in the near future by AI. But humans will be needed to do new things.
Take the Hugging Face incident. Why did it happen? Because the people whose task was to set up a testing framework took shortcuts. Why did they? Because there weren't enough people who were assigned to do the job. Why not? Because the job is too new and not enough people are qualified to do it. It's a job that simply did not exist 3 years ago. But 3 years from now, this job might very well employ tens of thousands of high skill knowledge workers.
Oh for sure, this was a simplistic analysis assuming AI adoption caps out at some X% of job responsibilities where X << 100%.
Unfortunately, I fear that may not be the most likely outcome. I've posted some comments on this before, but when I start thinking about how deeply everything will change once people figure out how to properly leverage AI, I see no outcome other than significant, widespread job losses.
As you indicated, at that point we will have much a bigger problem than the valuation of the AI industry. I'm not sure how it will get solved, I just know it will HAVE to be, because it would be an existential problem for everybody: people, governments, even the billionaires! Because now consider the 3rd order effects: if nobody can buy the stuff that's produced, how can billionaires get even richer? ;-)
fittingopposite 3 hours ago [-]
> It's reasonable to assume that if AI drop-in-replaced all those knowledge workers, AI companies could credibly charge somewhere in that order of magnitude, because that's what the market is already bearing.
Future supply and demand will set the price - not what is paid today. If supply by open models is vast and cheap, I can't see that the entire knowledge industry can hold the current size. It'll rather collapse to a fraction of its current value.
flyinglizard 7 hours ago [-]
You’re right; given that most of the money in the AI market is injected through OpenAI and Anthropic (which collect it through both selling equity and through customer revenue), the 7-8T is just a derivative of that.
melvinroest 17 minutes ago [-]
Yea I get the bearishness from my own personal experience.
Personally, I use LLMs for a lot of things. Oftentimes, I'm a think out loud type of person so even having something that feels like a rubber duck, but more competent, is already amazing for me. And LLMs are a lot more competent than a rubber duck.
But especially sometimes I've noticed that LLMs can be unbelievably stupid. It recently happened a few times with Fable 5.1 as well. Ultimately, I think it comes down to that LLMs can't think broadly. In software development one can usually see this too. For example, a whole app might be built by an LLM and it didn't spend a single token thinking about security because the prompter is at the level of "build a dating app for dogs, make no mistakes". Now you have a dating app for dogs that is insecure.
Since I prompt for almost everything in my life to have an LLM as a sounding board, I'm usually not an expert either. I've noticed LLMs are amazing at "bulk search engine information aggregation" (or whatever you want to call it). So if I need something from the Dutch government, I can find it way more quickly. But oftentimes I've noticed that going for a walk and thinking about a particular thing I'm facing is a more effective way of finding a good solution.
Other times times they are not incredibly stupid, but can't form a strong opinion. This usually happens when I'm tackling a wicked problem [1]. When that's the case, prepare for LLMs to sway with you for every small change in your opinion that you ever will experience.
So I agree: drop in replacement for knowledge workers? No. Rigorous specification is usually needed yes. Though, the small win here is that it doesn't always need to be as rigorous as programming is and it can happen in natural language. It depends on the topic/problem being tackled.
I really like them as UX tools though. Amazing for interactive prototyping and requirements elicitation. And that also corresponds with what the author is saying. Though I find it a bit of a disservice saying "just 3". You know how hard requirements elicitation is? It became a whole lot easier thanks to LLMs (I might change this opinion in a year, haha, but this is the opinion I hold now).
Short and to the point! Open and cheap models will undercut the big labs continuously. The blast radius won't be pretty once spending commitments knock the door.
woeirua 7 hours ago [-]
Open models wont be open for long. No one is going to release an open model capable of chaining zero-days. Even the Chinese aren't that reckless because it will just be turned around and used against them.
danny_codes 4 hours ago [-]
As compute prices fall it gets easier and easier to make "frontier" models. So it's inevitable that commodity, open source models of equivalent capacity to today's "frontier" models will be available to the public. Remember this is just weights, anyone can download them and run it whenever they like. The only constraint is compute.
Wazzymandias 4 hours ago [-]
depends on the blast radius of zero-days, it's not like there's a continuous immediate release process for these models; they can eval internally before releasing publicly
ransom1538 6 hours ago [-]
I haven't heard of the term "chaining zero-days". Now as a SRE I wont sleep.
pvab3 8 hours ago [-]
I agree with you but I'm still worried about the safety of open weight models as well. Both aligned and unaligned models.
m3kw9 7 hours ago [-]
You said it like labs like open AI doesn’t know and don’t constantly make moves to prevent that undercutting
bluegatty 6 hours ago [-]
"are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers, "
No, they're really not.
They're priced in a way that would imply AI will be universal form of compute, alongside traditional deterministic systems - which it will be.
And that they will capture most of that ... which they won't.
The Frontier Labs are a very bad buy at a high price, but that partly has to do with wacky pricing, but actually mostly has to do with their relatively weak place in the value chain.
The money is going to Nvidia, who have the most powerful position.
A bit like how a retailer can take all the margins of some innovative product, if they own the channel.
AI is over-hyped, the Frontier Labs are over priced - but AI is here to stay, and will grow. Not like Skynet, but like a new form of compute. And it will take it's time, and the profits will be reaped by those with the power.
lukewarm707 6 hours ago [-]
ai has a >10% chance of causing human extinction, according to anthropic big heads.
if that's true, you are wrong.
if that's false, anthropic is dishonest. why trust a dishonest company to be worth anything?
bluegatty 5 hours ago [-]
I think that the AI people believe in their own nonsense a bit.
Like - the guy on TV talking about 'AI will destroy everything' ... I don't think he's lying.
I think they are like we here on HN and Reddit and a bit caught up in our own thoughts.
If AI were unleashed, in raw form today, it could cause havoc.
Bad. Maybe very bad but I think we'd get over it.
It would probably trigger a recession (because we are in a bubble - it would pop it), and people would 'blame the AI' for sure.
But it would be a bit dot-com ish kind of recession.
The amplifiers would be geopolitical instability.
krapp 4 hours ago [-]
>If AI were unleashed, in raw form today, it could cause havoc.
What is "raw form?"
bluegatty 3 hours ago [-]
The SOTA models are heavily 'guardrailed' today; they won't let you do all sorts of things.
randomImmigrant 8 hours ago [-]
I think bearish on LLMs for automation, and bullish for LLM+human experts in specific fields, is about the right expectation for current architectures.
Apart from issues with task generalization, or perhaps related to it, is the fact that LLMs have real trouble with timekeeping, and cannot estimate the real world time it will take them to do things very well. This plus the memory issues make dreams of long horizon agents, that could plausibly handle changing specifications, quite implausible with current architectures.
In narrow domains with more deterministic outputs though, this is less of an issue, and we see multiple agents succeed much better.
The fusion of that capacity, with humans in the loop able to better direct such agents and act as their temporal tethers, is where I think the real action will be for a while at least.
handfuloflight 7 hours ago [-]
> This plus the memory issues make dreams of long horizon agents, that could plausibly handle changing specifications, quite implausible with current architectures.
Any reason why that can't be solved through context management and keep-forward scaffolding?
arm32 5 hours ago [-]
Write the same sentence you just wrote back to me, but in only four words and let’s see if it has the same meaning.
handfuloflight 5 hours ago [-]
Remember specifics, keep going.
lantry 5 hours ago [-]
"Any reason why that can't be solved through context management and keep-forward scaffolding?"
becomes
"load bearing context seam"
/s
ausbah 9 hours ago [-]
> the best alternative to rigorous specification is human review. human review doesn't scale well to the volumes of output produced by language models. to make matters worse
when the business model is selling more tokens you get such per serve ice times that lead to “more” thinking, engagement baiting, fluffy narratives, and straight up dark patterns
robinpie 9 hours ago [-]
I really appreciate seeing a tempered take that's not literally denialist about current capabilities.
an0malous 8 hours ago [-]
I don’t know who you’re talking about, even the most bearish people like Gary Marcus and Ed Zitron acknowledge that LLMs are useful in these same cases the OP admits. Gary Marcus is even still a long term AI advocate, he just doesn’t think LLMs are enough and we need more foundational breakthroughs. Zitron says it’s valuable technology but not worth the trillion dollar valuations the frontier labs are claiming.
The lack of temperament is very skewed towards the bulls who have been saying AGI is here, software engineering is solved, mathematics is solved, it’s going to destroy the white collar job market, and it’s going to kill us all for like 5 years now.
pvab3 8 hours ago [-]
Even a lot of the people who think that LLMs are a dead end think that we will soon find something signficantly more powerful, which I find deeply alarming. I don't want to know what my white-collar knowledge work will look like in a decade or 2.
arctic-true 8 hours ago [-]
Gary Marcus is an especially puzzling addition. If I recall correctly, he has made statements along the lines that superintelligence this century is more likely than not. If you’re AGI-pilled that might read as bearish, but that is still extremely rapid progress in the grand scheme of things.
brindleth 8 hours ago [-]
> current frontier models need laborious oversight and guardrails on even the simplest tasks
It is literally denialist about current capabilities
jaykru 8 hours ago [-]
why don't anthropic and openai ship yolo mode by default?
Human-Cabbage 8 hours ago [-]
They do…? Well, “auto” mode has been default in Claude Code for a couple months now. It’s effectively “safer yolo:” tool calls are inspected by a separate classification system (another smaller LLM, I believe) to approve or deny. And you can always layer on additional sandboxing mechanisms to limit the blast radius deterministically.
vmg12 8 hours ago [-]
> They do…? Well, “auto” mode has been default in Claude Code for a couple months now
They have never shipped "yolo" mode by default. Auto mode is not yolo mode. They trained a task specific model just for ensuring the llm didn't accidentally delete every file from your computer.
SyneRyder 8 hours ago [-]
Anthropic basically does at this point with Auto Mode being default. Or was that the point you were making?
jaykru 8 hours ago [-]
That is the point I was making, that auto mode is itself a guardrail on top of the model (and not a perfect one.) auto mode seems to cover merely actions the model could take that are clearly bad, like wiping your disk, using an overly privileged context to complete the task, etc.
I recently tasked a GPT model in Codex with implementing part of a new architecture I'm working on. I gave it a very detailed spec and the code it produced looked pretty reasonable and passed my tests. It even did exceptionally well in my evals, so I excitedly declared victory to a few friends. The next day after more careful review I found that the architecture implementation was totally correct, but the model had slipped a one line change to the observation encoding of the RL environment I was prototyping against. The encoding change made the learning problem essentially trivial; the architecture itself, I later realized, had a major flaw that was revealed by returning to the natural encoding.
This is the type of reward hack that is hard to paper over with easy guardrails like auto mode and even harder to specify out. It's also the type of thing a reasonable human wouldn't do unless they were intentionally trying to deceive you.
jaykru 9 hours ago [-]
Thanks :) I do enjoy and use these things every day and the current capabilities are indeed amazing, just ludicrously overpriced at the frontier.
dumberquestions 8 hours ago [-]
I can see current limitations, but how do you expect capabilities to change in the next few years? A repeat of the gain that happened in the last two years feels like it would be significant, even if it took a little more than two years this time around.
yunwal 7 hours ago [-]
> those who need done a small set of narrowly defined tasks with existing clear guardrails: repetitive physical labor in a controlled environment, call center and customer service chat work, etc.
I have no idea how people can so confidently say that call center work is a “controlled environment” or “repetitive”. It’s almost by definition not repetitive or controlled. Customer support is what I go to when the controlled environment has failed
vachina 6 hours ago [-]
Depends on what customer support means.
Typically it means knowledge retrieval from a KB or manipulating a control surface not visible to you.
fhe 4 hours ago [-]
came here to say exactly this. in fact, this is probably why we are not seeing a lot of AI application on customer service use case, and when we see one, it's almost always frustrating.
jumploops 7 hours ago [-]
LLMs are basically multi-dimensional magic mirrors.
Depending on where you point them, they can be incredibly useful.
They can even be useful when you point them at each other (though increasingly difficult to get good results).
I'm excited for the promise of RSI and a future where models have inherently "live" weights, but it's not clear to me that the transformer is more than a useful tool to help us get there.
againstapples 8 hours ago [-]
> the models generalize well only on tasks within a small neighborhood of the specific tasks they've been trained on, and even then with severe caveats. the frontier labs have developed a general recipe to teach models almost any specific task enjoying clearly defined levels of task performance; many tasks are covered in the training data
Is this really any different to how humans learn, it takes a lot of training on one specific task to make a human expert as well?
bravoetch 8 hours ago [-]
I was a young child when I learned chess by reading a short book, then practicing with a friend. That is not how LLMs learn. I'm no expert on LLMs, but if you showed a human all chess games and books in all history and then said 'play chess' and they still kept making illegal moves, they would have to have a brain injury.
harimau777 8 hours ago [-]
It seems like an LLM potentially could learn that way if each practice game it participated in was added to its training data.
willmarch 7 hours ago [-]
Yes, this is essentially how AlphaGo and AlphaZero algorithms work to train superhuman Go/chess/shogi agents. It’s an elegant algorithm that is analogous to how humans learn games.
zug_zug 5 hours ago [-]
Well except AlphaZero played 44 million chess games in that time (and actually played with a 44 core computer). So I'd like to point out that the human is still just a few orders of magnitude more efficient.
willmarch 5 hours ago [-]
Yes, we all know that biological systems are more efficient than machines through billions of years of evolution and natural selection but the overall process is largely the same (interacting with an environment, learning from results, improving underlying architecture, etc); efficiencies will come with more time and improvements.
fhe 4 hours ago [-]
but maybe... the said human has also read every other piece of text ever written, including ones about other (similar?) board games, which in aggregate vastly dwarfs whatever he has read on chess, that non-chess reading could have corrupted whatever he's learned about chess?
danielmarkbruce 8 hours ago [-]
Humans wear a lot of hats when the do work. They don't even realize how many. My experience with building real systems using LLMs is that you have to be very explicit about such hats and you don't realize how many are worn until you see edge case after edge case after edge case. Check this. Check that. Check this. Check that. Check check check.
JohnMakin 8 hours ago [-]
> Is this really any different to how humans learn
yes.
knuppar 8 hours ago [-]
being a bit more specific: the sample efficiency of humans is orders of magnitude larger for more abstract concepts. the same doesn't hold for memory-intensive tasks though (like any kind of trivia), but that only takes you so far.
danpalmer 6 hours ago [-]
We've had technology beating humans on memory for millennia, and we've had technology beating humans on computation for many decades now.
The tricky thing with LLMs is describing what they actually do. They are too clearly beating humans on some things, but what exactly? Memory – already done, they're bad at basic computation (all LLMs just write code for actual computation/calculation). And as you say, they do badly at more abstract concepts.
bananzamba 8 hours ago [-]
Also doesn't the very good ARC AGI 2 score of GPT-6 Astra kinda contradict this, since each problem is its own game with very different rules
someguynamedq 7 hours ago [-]
> current frontier models need laborious oversight and guardrails on even the simplest task
As models advance, we shift the goalpost for what "simplest task" means. Before, "simplest task " meant "write a coherent English sentence." Now, "simplest task" means autonomously fix, review, and merge a bugfix.
abeppu 6 hours ago [-]
Eliza wrote coherent English sentences.
vatsachak 8 hours ago [-]
I agree with the caveat that it's more like a cracked junior engineer who can manage swarms of interns.
Frontier Labs will probably survive off hype valuations but will serve the important purpose of discovering architectures/techniques that will probably spread through rumors/transfers to the rest of the world.
slibhb 6 hours ago [-]
> the frontier labs are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers
That's a reason to be bearish about AI companies, not LLMs. But is it even true? OpenAI and Anthropic have each reported ~50 billion in revenue with ~900 billion valuations. That's a high ratio but I'm not sure if follows that the only way it pans out is if we get "fully automated drop-in replacement for most knowledge workers".
It wouldn't shock me to see those revenue numbers scaling up to where they need to be over the next decade ( to, say, ~400 billion) without ever achieving drop-in worker replacements.
zug_zug 5 hours ago [-]
I looked at the math and I think it's true. Remember revenue is just sales, not profit. These labs are shooting for > $1T valuations, which traditionally means your PROFIT is at least 1/20th or 1/30th of that (so let's say minimum 30B$/year PROFIT).
These companies however are LOSING money (anthropic tries to make it sound like it's profit by deviating from accepted accounting principles) and subsidizing these models. When accounting for all the engineering salaries, training, GPUs, etc, what's their best-case realistic margin three years out, 10%?
So to we'd need a scenario where companies are spending a collective 300B annually on AI (believable) but ALSO that these companies jack up their margins WITHOUT companies switching to the cheaper open-source models (even when there's a $300B incentive to do so).
alain94040 7 hours ago [-]
Not convinced by those points.
In particular, I found this very misleading or irrelevant:
a typical CPU project anecdotally has about three times as many specification and validation engineers as design engineers and a 5:1 ratio is not unheard of
The reason silicon design has such verification to design ratio is because the cost of one bug is many, many orders of magnitude higher than software. Both in dollar cost and in schedule cost (it takes months to fab a chip, and if you messed up and need to spin a fix, it costs tens of millions of dollars, not counting any design engineering cost).
I don't think you can extrapolate these very industry-specific facts to judging LLMs.
danpalmer 6 hours ago [-]
> The reason ... is because the cost of one bug is many, many orders of magnitude higher than software. Both in dollar cost and in schedule cost (it takes months ... and if you messed up and need to spin a fix, it costs tens of millions of dollars, not counting any design engineering cost).
Aren't you just describing waterfall? That's still very prevalent in software engineering, and pretty much any other type of engineering – civil, chemical, building, architecture, drug discovery.
It's typically true that software can fail faster and cheaper, but it's also true that the costs are still vastly higher to fix later in the process.
alain94040 6 hours ago [-]
No. Silicon is on another level. Which is why the EDA verification is an industry on its own.
Sure, there are some software that have similar "can't have bugs" requirements. I imagine the computers on Moon missions also had that kind of high bar. I wouldn't use NASA requirements as a proof for how LLMs should be used.
pfdietz 9 hours ago [-]
Specifically: bearish on LLMs generally, not bearish on LLMs for pure math.
jaykru 9 hours ago [-]
yes, huge for pure math and activities that look like it.
danielmarkbruce 8 hours ago [-]
Doesn't really even need to look like it. If you can verify rewards, RLVR will optimize really really well. If you can't... it's a struggle. There are probably fewer fields where you can verify rewards than one might hope.
skydhash 7 hours ago [-]
> There are probably fewer fields where you can verify rewards than one might hope.
2 tasks I've done today that I believe robots are nowhere near being able to do: Cleaning my wardrobe and draining bad fuel out of my generator. As in generic use cases.
camd32 6 hours ago [-]
> current frontier models need laborious oversight and guardrails on even the simplest tasks.
This is only true if you are concerned about the intermediate steps of the model as opposed to the outcome. The huggingface hack was a perfect example of the model doing whatever it takes to accomplish the goal of maximizing its score.
willy_k 4 hours ago [-]
So, if you are concerned about what the model does? Yeah.
Founderarcstone 8 hours ago [-]
I am bullish on AI. At some point well see some true advancements.
aogaili 8 hours ago [-]
good post/take.
war-is-peace 8 hours ago [-]
refreshing to see amongst the endless tide of "i haven't written a single piece of code since 2025, llms are so good that they have already replaced everyone" gaslighting
- i also have no idea what everyone and their momma on HN is running for more than 5 mins in the name of "agentic AI"
stogot 5 hours ago [-]
Won’t this change though?
> the present problem of reward hacking can be solved only by rigorous specification by domain experts. the time of domain experts is expensive. rigorous specification is itself a skill, demanding its own expertise outside of a given problem domain. even many skilled software engineers are bad at it. for the vast majority of domains, the intersection of domain experts and specification experts is ludicrously small.
moomoo11 6 hours ago [-]
the issue most of you seem to not realize is that when you put these models in a loop, you are able to do more and more insane and cool things.
have you guys actually designed, built, and deployed agentic workflows?
it is actually quite hard, requires tons of time spent on evals and testing to ensure accuracy, but when it starts to work it is mind blowing.
there is no going back.
listening to people yap about AI when they have only surface level or one dimensional exposure to LLMs and "AI", but have not actually put innovations to work IN PRACTICE.. is a waste of time
zzzeek 7 hours ago [-]
great, autonomous LLMs will fail. that's actually perfect. they work amazingly well when we're telling them what to do. no autonomy needed, no destruction of humanity. that's all win
DO you know those ice cream shops that sell 30 different flavours. Everybody likes a different flavour, some people dont even like ice cream and buy nothing.
Some people will complain about the wrong flavours, or missing flavours, or the price, the long lines or maybe it closes early on fridays.
Summarise means different things to different people.
willy_k 3 hours ago [-]
Thats a browser level task.
Rendered at 07:28:11 GMT+0000 (Coordinated Universal Time) with Vercel.
https://arxiv.org/html/2509.24239v4
Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.
The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.
> Gemini 2.5 Pro, O3, Claude Sonnet 3.7 and ChatGPT 4.1
The gap in capabilities between those models which they tested, and actual current frontier ones is enormous. I would not trust that any conclusions they made are applicable.
https://chessbench-ai.github.io/#leaderboard
It's also worth noting that the very latest models (GPT-6 and Fable 5.1) actually play worse than their immediate predecessors, so it is likely that the labs are not benchmaxxing for this yet. If they did, I'm sure they could come up with something superior to humans. But there is probably very little demand for this compared to IT stuff.
> About their ELO ratings from their own website:
> A field-relative rating calculated within ChessBench. It compares performance among the tested models and is not a direct equivalent of a human chess rating.
I am around 1600 elo in over the board I can mop up Astra Fable etc even if I give them literal infinite time and all the subagents and internet access..
Please folks at least use your AIs to read stuff before making claims.
AI is not GM level, it's not even 1600, I am 1600 by using memorized openings people frequently fall for with very basic intuitions.
A GM is 2600 they can beat me in under 20 moves...
Why do I even scroll through this website. For a moment I truly felt fooled, but then I read like a human should.
Maybe I should stop doing that will be a happier life, don't think just believe in the AGI.
Don't use the word infinite in any CS claims. They can recreate or approximate monte Carlo tree search and it technically is still a correct solution in your framing of the problem so long they defeat you.
But given how easily I can crush them and how often they want to make illegal moves (btw above bench seems to use a harness that pokea the model until it gives valid moves).
I would rate them around 500-800 big range but at that level it's all about if the model can recall an opening or not. If it plays good first 4-8 moves the person on the end will fumble for certain and they win.
I can play good/best moves till 14-15 moves if I remember the lines and find someone who falls for it.
If you could give them the lines as prompts like the best 20-30 openings then they will be around 700-800.
700 is around the rating for a human who doesn't know the tricks but can do bare minimum calculations and understands the rules thoroughly.
Anyone who casually plays on a regular basis can beat them more often than they lose. As you said if you just know the core openings (and end games, both of which you can get a handle on with modest effort) you will generally win.
Edit: reminder we had computers beating the best players in the world literally decades ago. LLM’s are remarkable tools but the current promises and expectations are ridiculous
In some ways this is reflective of the AI experience at large, sometimes shockingly competent but then also sometimes ludicrously incompetent.
You're thinking about this the wrong way. The system is built and delivered as it is because that's how the providers make the most money. If they cared to have it perform well in chess games, you'd see a different shape and behavior.
We shouldn't ask the multibillion dollar automated software generation system to play games with us any more than we should ask a Boeing's flight guidance system to do so.
>If they cared to have it perform well in chess games, you'd see a different shape and behavior.
So the things they claim are on the verge of AGI actually aren’t? They need to be trained for specific tasks?
> The system is built and delivered as it is because that's how the providers make the most money. If they cared to have it perform well in chess games, you'd see a different shape and behavior.
These comments indicate a complete failure to understand the technology.
I won't respond again.
Delusion runs deep in HN circles.
I say that as someone heavily invested in AI startups and projects and as someone working in the field.
I think most people on HN should touch grass and find real human contact. Lmao
Incredible reasoning all around here.
Also AI bros: LLM can’t beat an avg chess player. But that doesn’t mean anything. It doesn’t count
Why should that matter?
Because we want to use this as a replacement for humans, and the average human can learn the rules of chess without needing to see the rules explained hundreds of thousands of times in millions of games.
So, yeah, it matters if a model has millions of examples of something in its training set and still cannot follow the rules.
So we have a situation where very powerful and influential people are saying we will have AGI in 6 months (if we don’t already), yet the facts on the ground are so clearly pointing in the opposite direction.
People are holding it wrong, deliberately or not. Some are inventing bad faith measures so they can claim AI sucks.
We all know AI can code, but the question it all stemmed from what if it's AGI or GM level in chess on it's own.
You can't just back pedal from the statement that apparently being able to code a chess engine is the same as being good at chess.
I can write a chess engine that beats Magnus Carlson without AI that alone neither makes me GM level or AGI or any of the other claims the above comments seem to be making?
He definitely needs to touch grass.
Y'all seem to miss the point of this forum. Building and hacking and science and engineering.
I swear there's a whole lot of you who just like to look down instead of up. There's a whole universe up there.
Is code omnipotent, I have been in software all my life and I would hard agree here.
Sure stuff LLMs can do with being good at parts of code reproduction is incredible. And honestly it's the new way to do a lot of things but I have not see an iota of proof that it can scale across the board.
For instance Maths is just code with different symbols and slightly less universally legible concepts.
AI is the best invention at figuring out or walking the search space and directionally doing logically computation over general software adjacent stuff.
But that's it, I am certain a bunch of companies will make a lot of money despite no AGI.
I think people either don't understand AGI or don't understand how real world works.
Until an LLM can bow it's head take responsibility for mistakes made and ensure they aren't repeated again with 100% confidence to the leadership it's inarguably a tool a rather questionable one at that.
As a 1500 elo human I can tell you that a 1500 elo chess engine doesn't play like anything like a 1500 elo human.
GPT-6 almost never suggests an illegal move anymore while even Sol still did so time to time
They are of course "wrong" if you don't read the faint fine print and sensibly interpret them as FIDE or similar ratings.
"Elo is relative to the ChessBench field."
> "current frontier models need laborious oversight and guardrails on even the simplest tasks"
I feel this statement is extreme. I can't personally reconcile it with any of the projects we're regularly seeing get delivered largely by LLMs now.
What are you thoughts? Like, what's your position here? Even if you sincerely believe frontier models need laborious oversight on even the simplest of tasks, do you think that accurately captures and reflects the current state and progress of frontier LLMs?
Don't get me wrong, there's lots of things LLMs can't do well, but the idea that they're basically not helpful for even the simplest of tasks seems... disingenuous?
2. Chess seems to be a poor benchmark for generalized strategic reasoning. People who are good at it rely more on experience and deep domain expertise than on skills that generalize to make them experts at unrelated tasks.
3. The study sounds like proving humans will never fly because they don’t have wings. In reality, humans do fly, and Claude Fable would destroy any human at chess by coding a strong enough engine on the fly.
The idea that anything other than a breathless blog post about the latest model snapshot is useless is really poisonous to proper debate on AI issues
also I think the operative letter in AGI is the G - and if the G is short for 'variably competent savant-like hyperfocus on certain kinds of software coding and not any other general skill' then its not really G at all, is it?
A bash script can clone and build stockfish, feed in human moves, and reply. By your standard, this bash script would "destroy any human at chess."
Are you interested in assessing the intelligence of the model, or the intelligence of the tools the model can use?
People are good are 1900 or 2100 above and the top ones who spend decades in the field i.e. deep expertise are well in the 2200-2700 range.
A 1100 player is none of these things, they are purely relying on strategic reasoning there is a good chance they cannot name a single opening or articulate clearly why a move was appropriate. 1100 is quite low bar.
Delusional, but then Claude fable also isn’t beating any human at chess, the engine is.
Yes some people can do it but most people can't even if they're unusually intelligent.
You really need to be giving the LLM a board representation.
But nobody wants that.
There's more to games than simply winning you know.
The caveat is: It depends on the task.
Are there reams of chess moves that the model can train off of? No.
Are there reams of math papers the model can train off of? Yes.
I think the line of criticism around LLMs sucking at chess makes more sense when you understand what the AI companies are saying about the future trajectory of these models.
The entire recursive self improvement story falls apart once you point out that there is not much "cross domain transfer learning". Meaning that training an LLM to become good at coding, math, etc, will eventually transfer into them being good at other skills that were not explicitly trained for.
Using games like chess which have little economic value is actually a good test for this. What's even more surprising about them sucking at chess is how much information about chess strategy exists in the training data.
This is as false as something can possibly be. There are open databases of millions of chess games spanning hundreds of years.
The only reason LLMs are this bad at chess is because the labs don’t care about chess performance so they’re not going out of their way to train the models for it. The ability they do have is from what chess information happens to be in the training data, plus whatever general reasoning abilities they may be able to apply.
For real??
If the models were actually intelligent, the way that the boosters claim, they wouldn't need to be tuned to play chess in order to be good at it. That's kind of the point of intelligence, that it is generically applicable to whichever task one wishes.
Your assumptions/intuition about generic human intelligence feels quite incorrect, considering LLMs currently play better than a brand new human player would (presumably without any attempt to fine tune them specific on chess, such as playing thousands of games).
They’ve ingested all the literature on playing chess, a brand new human player has not.
We seem to be moving goalposts to the point that humans don’t even live up to the expectations of the AI critics. The only way you get better at chess is by playing a lot of games and learning from mistakes, that goes for humans or AI agents, not simply by reading about chess.
How can you play without being aware of the rules and how can you learn from your mistakes without knowing they are mistakes? That’s what I said about reading a book of two. It is to kickstart the process. Then mastery is gained over time through practice.
This kickstarting then gradual refinement is how most people learn. And the foundational knowledge stays. Even a basic player knows to not do illegal moves.
It feels like you're trying to say that humans never make illegal moves while learning chess, which doesn't match with my experience. I'm trying to understand your overall point.
Coding is a matter of translating the natural language description of a problem to the code specification while keeping the semantics fixed (and imputing the unspecified semantics as necessary). It is not considerably more difficult than translating between two dissimilar natural languages. Chess isn't a matter of language translation, but a compute heavy game of finding the best move out of many possibilities with wide variation in the quality of each move. Chess takes directed practice and reinforcement whereas language translation does not.
People think that if one mention exists in the training set, then the LLM is perfect at it.
It also makes one wonder: What could AI solve if we managed to orchestrate billions worth of agents to take on a specific problem?
IMO the very best case scenario / potential for these are likely better than we think, but right now hidden due to logistical and financial reasons.
But if we assume that the model costs will continue to drop by a factor of 5-10 annually, there will always be a latency of a couple of years between what is completely out of reach, and what is financially viable.
Basically: If you knew AI could be affordable enough in 3-5 years so that even the most underfunded researchers could use it to solve cancer, how much would you value it now?
> the frontier labs are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers...
Even assuming this is how the AI companies are being valued (they're not), the numbers are off.
The "value" of most knowledge workers -- based on what enterprises currently pay for them -- is $50 - 70 trillion annually. It's reasonable to assume that if AI drop-in-replaced all those knowledge workers, AI companies could credibly charge somewhere in that order of magnitude, because that's what the market is already bearing.
So if their hypothetical revenues are double-digit trillions and valuations are some multiple of that, the entire AI industry would be valued at double-digit trillions at the least.
Yet cumulatively the industry (the frontier labs + the SWAG estimate of the AI parts of all the other players) are valued at, say, ~6 - 7 trillion? Which seems like a fair approximation of how much knowledge work they can currently automate.
What do you mean? The sum of ALL US salaries is $13.4 Trillion per year. According to google $65T is the sum of ALL salaries Globally (not just knowledge workers). It's not reasonable to assume AI is a drop-in-replacement for any job yet (perhaps bottom tier customer support from oversees?).
> So if their hypothetical revenues are double-digit trillions and valuations are some multiple of that
So you're sort of premising here than more than 16% or 1/6 of all the world's jobs get replaced by AI. Hopefully you can understand that's both not the current AI capability and also would be a terrible (unprecedented?) economic shock.
Unfortunately, I do fear that AI adoption will go beyond augmentation to automation, and I do fear an economic shock. Just posted this down-thread: https://news.ycombinator.com/item?id=49722616
And then there’s the second order effect: if all the knowledge workers get automated, who is going to buy the stuff that’s produced?
Take the Hugging Face incident. Why did it happen? Because the people whose task was to set up a testing framework took shortcuts. Why did they? Because there weren't enough people who were assigned to do the job. Why not? Because the job is too new and not enough people are qualified to do it. It's a job that simply did not exist 3 years ago. But 3 years from now, this job might very well employ tens of thousands of high skill knowledge workers.
[1] https://en.wikipedia.org/wiki/Lump_of_labour_fallacy
Unfortunately, I fear that may not be the most likely outcome. I've posted some comments on this before, but when I start thinking about how deeply everything will change once people figure out how to properly leverage AI, I see no outcome other than significant, widespread job losses.
As you indicated, at that point we will have much a bigger problem than the valuation of the AI industry. I'm not sure how it will get solved, I just know it will HAVE to be, because it would be an existential problem for everybody: people, governments, even the billionaires! Because now consider the 3rd order effects: if nobody can buy the stuff that's produced, how can billionaires get even richer? ;-)
Future supply and demand will set the price - not what is paid today. If supply by open models is vast and cheap, I can't see that the entire knowledge industry can hold the current size. It'll rather collapse to a fraction of its current value.
Personally, I use LLMs for a lot of things. Oftentimes, I'm a think out loud type of person so even having something that feels like a rubber duck, but more competent, is already amazing for me. And LLMs are a lot more competent than a rubber duck.
But especially sometimes I've noticed that LLMs can be unbelievably stupid. It recently happened a few times with Fable 5.1 as well. Ultimately, I think it comes down to that LLMs can't think broadly. In software development one can usually see this too. For example, a whole app might be built by an LLM and it didn't spend a single token thinking about security because the prompter is at the level of "build a dating app for dogs, make no mistakes". Now you have a dating app for dogs that is insecure.
Since I prompt for almost everything in my life to have an LLM as a sounding board, I'm usually not an expert either. I've noticed LLMs are amazing at "bulk search engine information aggregation" (or whatever you want to call it). So if I need something from the Dutch government, I can find it way more quickly. But oftentimes I've noticed that going for a walk and thinking about a particular thing I'm facing is a more effective way of finding a good solution.
Other times times they are not incredibly stupid, but can't form a strong opinion. This usually happens when I'm tackling a wicked problem [1]. When that's the case, prepare for LLMs to sway with you for every small change in your opinion that you ever will experience.
So I agree: drop in replacement for knowledge workers? No. Rigorous specification is usually needed yes. Though, the small win here is that it doesn't always need to be as rigorous as programming is and it can happen in natural language. It depends on the topic/problem being tackled.
I really like them as UX tools though. Amazing for interactive prototyping and requirements elicitation. And that also corresponds with what the author is saying. Though I find it a bit of a disservice saying "just 3". You know how hard requirements elicitation is? It became a whole lot easier thanks to LLMs (I might change this opinion in a year, haha, but this is the opinion I hold now).
[1] https://en.wikipedia.org/wiki/Wicked_problem
No, they're really not.
They're priced in a way that would imply AI will be universal form of compute, alongside traditional deterministic systems - which it will be.
And that they will capture most of that ... which they won't.
The Frontier Labs are a very bad buy at a high price, but that partly has to do with wacky pricing, but actually mostly has to do with their relatively weak place in the value chain.
The money is going to Nvidia, who have the most powerful position.
A bit like how a retailer can take all the margins of some innovative product, if they own the channel.
AI is over-hyped, the Frontier Labs are over priced - but AI is here to stay, and will grow. Not like Skynet, but like a new form of compute. And it will take it's time, and the profits will be reaped by those with the power.
if that's true, you are wrong.
if that's false, anthropic is dishonest. why trust a dishonest company to be worth anything?
Like - the guy on TV talking about 'AI will destroy everything' ... I don't think he's lying.
I think they are like we here on HN and Reddit and a bit caught up in our own thoughts.
If AI were unleashed, in raw form today, it could cause havoc.
Bad. Maybe very bad but I think we'd get over it.
It would probably trigger a recession (because we are in a bubble - it would pop it), and people would 'blame the AI' for sure.
But it would be a bit dot-com ish kind of recession.
The amplifiers would be geopolitical instability.
What is "raw form?"
Apart from issues with task generalization, or perhaps related to it, is the fact that LLMs have real trouble with timekeeping, and cannot estimate the real world time it will take them to do things very well. This plus the memory issues make dreams of long horizon agents, that could plausibly handle changing specifications, quite implausible with current architectures.
In narrow domains with more deterministic outputs though, this is less of an issue, and we see multiple agents succeed much better.
The fusion of that capacity, with humans in the loop able to better direct such agents and act as their temporal tethers, is where I think the real action will be for a while at least.
Any reason why that can't be solved through context management and keep-forward scaffolding?
becomes
"load bearing context seam"
/s
when the business model is selling more tokens you get such per serve ice times that lead to “more” thinking, engagement baiting, fluffy narratives, and straight up dark patterns
The lack of temperament is very skewed towards the bulls who have been saying AGI is here, software engineering is solved, mathematics is solved, it’s going to destroy the white collar job market, and it’s going to kill us all for like 5 years now.
It is literally denialist about current capabilities
They have never shipped "yolo" mode by default. Auto mode is not yolo mode. They trained a task specific model just for ensuring the llm didn't accidentally delete every file from your computer.
I recently tasked a GPT model in Codex with implementing part of a new architecture I'm working on. I gave it a very detailed spec and the code it produced looked pretty reasonable and passed my tests. It even did exceptionally well in my evals, so I excitedly declared victory to a few friends. The next day after more careful review I found that the architecture implementation was totally correct, but the model had slipped a one line change to the observation encoding of the RL environment I was prototyping against. The encoding change made the learning problem essentially trivial; the architecture itself, I later realized, had a major flaw that was revealed by returning to the natural encoding.
This is the type of reward hack that is hard to paper over with easy guardrails like auto mode and even harder to specify out. It's also the type of thing a reasonable human wouldn't do unless they were intentionally trying to deceive you.
I have no idea how people can so confidently say that call center work is a “controlled environment” or “repetitive”. It’s almost by definition not repetitive or controlled. Customer support is what I go to when the controlled environment has failed
Typically it means knowledge retrieval from a KB or manipulating a control surface not visible to you.
Depending on where you point them, they can be incredibly useful.
They can even be useful when you point them at each other (though increasingly difficult to get good results).
I'm excited for the promise of RSI and a future where models have inherently "live" weights, but it's not clear to me that the transformer is more than a useful tool to help us get there.
Is this really any different to how humans learn, it takes a lot of training on one specific task to make a human expert as well?
yes.
The tricky thing with LLMs is describing what they actually do. They are too clearly beating humans on some things, but what exactly? Memory – already done, they're bad at basic computation (all LLMs just write code for actual computation/calculation). And as you say, they do badly at more abstract concepts.
As models advance, we shift the goalpost for what "simplest task" means. Before, "simplest task " meant "write a coherent English sentence." Now, "simplest task" means autonomously fix, review, and merge a bugfix.
Frontier Labs will probably survive off hype valuations but will serve the important purpose of discovering architectures/techniques that will probably spread through rumors/transfers to the rest of the world.
That's a reason to be bearish about AI companies, not LLMs. But is it even true? OpenAI and Anthropic have each reported ~50 billion in revenue with ~900 billion valuations. That's a high ratio but I'm not sure if follows that the only way it pans out is if we get "fully automated drop-in replacement for most knowledge workers".
It wouldn't shock me to see those revenue numbers scaling up to where they need to be over the next decade ( to, say, ~400 billion) without ever achieving drop-in worker replacements.
These companies however are LOSING money (anthropic tries to make it sound like it's profit by deviating from accepted accounting principles) and subsidizing these models. When accounting for all the engineering salaries, training, GPUs, etc, what's their best-case realistic margin three years out, 10%?
So to we'd need a scenario where companies are spending a collective 300B annually on AI (believable) but ALSO that these companies jack up their margins WITHOUT companies switching to the cheaper open-source models (even when there's a $300B incentive to do so).
In particular, I found this very misleading or irrelevant:
a typical CPU project anecdotally has about three times as many specification and validation engineers as design engineers and a 5:1 ratio is not unheard of
The reason silicon design has such verification to design ratio is because the cost of one bug is many, many orders of magnitude higher than software. Both in dollar cost and in schedule cost (it takes months to fab a chip, and if you messed up and need to spin a fix, it costs tens of millions of dollars, not counting any design engineering cost).
I don't think you can extrapolate these very industry-specific facts to judging LLMs.
Aren't you just describing waterfall? That's still very prevalent in software engineering, and pretty much any other type of engineering – civil, chemical, building, architecture, drug discovery.
It's typically true that software can fail faster and cheaper, but it's also true that the costs are still vastly higher to fix later in the process.
Sure, there are some software that have similar "can't have bugs" requirements. I imagine the computers on Moon missions also had that kind of high bar. I wouldn't use NASA requirements as a proof for how LLMs should be used.
2 tasks I've done today that I believe robots are nowhere near being able to do: Cleaning my wardrobe and draining bad fuel out of my generator. As in generic use cases.
This is only true if you are concerned about the intermediate steps of the model as opposed to the outcome. The huggingface hack was a perfect example of the model doing whatever it takes to accomplish the goal of maximizing its score.
- i have no idea how anyone thinks the mighty next token predictor is going to eradicate diseases and eliminate poverty https://blog.florianherrengt.com/how-llms-work.html
- i also have no idea what everyone and their momma on HN is running for more than 5 mins in the name of "agentic AI"
> the present problem of reward hacking can be solved only by rigorous specification by domain experts. the time of domain experts is expensive. rigorous specification is itself a skill, demanding its own expertise outside of a given problem domain. even many skilled software engineers are bad at it. for the vast majority of domains, the intersection of domain experts and specification experts is ludicrously small.
have you guys actually designed, built, and deployed agentic workflows?
it is actually quite hard, requires tons of time spent on evals and testing to ensure accuracy, but when it starts to work it is mind blowing.
there is no going back.
listening to people yap about AI when they have only surface level or one dimensional exposure to LLMs and "AI", but have not actually put innovations to work IN PRACTICE.. is a waste of time
Some people will complain about the wrong flavours, or missing flavours, or the price, the long lines or maybe it closes early on fridays.
Summarise means different things to different people.