Even before this, Gemini 3 has always felt unbelievably 'general' for me.
It can beat Balatro (ante 8) with text description of the game alone[0]. Yeah, it's not an extremely difficult goal for humans, but considering:
1. It's an LLM, not something trained to play Balatro specifically
2. Most (probably >99.9%) players can't do that at the first attempt
3. I don't think there are many people who posted their Balatro playthroughs in text form online
I think it's a much stronger signal of its 'generalness' than ARC-AGI. By the way, Deepseek can't play Balatro at all.
Google has a library of millions of scanned books from their Google Books project that started in 2004. I think we have reason to believe that there are more than a few books about effectively playing different traditional card games in there, and that an LLM trained with that dataset could generalize to understand how to play Balatro from a text description.
Nonetheless I still think it's impressive that we have LLMs that can just do this now.
gilrain 7 minutes ago [-]
If it tried to play Balatro using knowledge of, e.g., poker, it would lose badly rather than win. Have you played?
ebiester 28 minutes ago [-]
It's trained on YouTube data. It's going to get roffle and drspectred at the very least.
winstonp 2 hours ago [-]
DeepSeek hasn't been SotA in at least 12 calendar months, which might as well be a decade in LLM years
cachius 2 hours ago [-]
What about Kimi and GLM?
zozbot234 22 minutes ago [-]
These are well behind the general state of the art (1yr or so), though they're arguably the best openly-available models.
dudisubekti 1 hours ago [-]
But... there's Deepseek v3.2 in your link (rank 7)
acid__ 59 minutes ago [-]
> Most (probably >99.9%) players can't do that at the first attempt
Eh, both myself and my partner did this. To be fair, we weren’t going in completely blind, and my partner hit a Legendary joker, but I think you might be slightly overstating the difficulty. I’m still impressed that Gemini did it.
littlestymaar 1 hours ago [-]
> . I don't think there are many people who posted their Balatro playthroughs in text form online
There are *tons* of balatro content on YouTube though, and it makes absolutely zero doubt that Google is using YouTube content to train their model.
sdwr 1 hours ago [-]
Yeah, or just the steam text guides would be a huge advantage.
I really doubt it's playing completely blind
Falsintio 1 hours ago [-]
[dead]
nubg 4 hours ago [-]
Weren't we barely scraping 1-10% on this with state of the art models a year ago and it was considered that this is the final boss, ie solve this and its almost AGI-like?
I ask because I cannot distinguish all the benchmarks by heart.
modeless 2 hours ago [-]
François Chollet, creator of ARC-AGI, has consistently said that solving the benchmark does not mean we have AGI. It has always been meant as a stepping stone to encourage progress in the correct direction rather than as an indicator of reaching the destination. That's why he is working on ARC-AGI-3 (to be released in a few weeks) and ARC-AGI-4.
His definition of reaching AGI, as I understand it, is when it becomes impossible to construct the next version of ARC-AGI because we can no longer find tasks that are feasible for normal humans but unsolved by AI.
Do opus 4.6 or gemini deep think really use test time adaptation ? How does it work in practice?
mapontosevenths 1 hours ago [-]
> His definition of reaching AGI, as I understand it, is when it becomes impossible to construct the next version of ARC-AGI because we can no longer find tasks that are feasible for normal humans but unsolved by AI.
That is the best definition I've yet to read. If something claims to be conscious and we can't prove it's not, we have no choice but to believe it.
Thats said, I'm reminded of the impossible voting tests they used to give black people to prevent them from voting. We dont ask nearly so much proof from a human, we take their word for it. On the few occasions we did ask for proof it inevitably led to horrific abuse.
Edit: The average human tested scores 60%. So the machines are already smarter on an individual basis than the average human.
WarmWash 32 minutes ago [-]
>because we can no longer find tasks that are feasible for normal humans but unsolved by AI.
"Answer "I don't know" if you don't know an answer to one of the questions"
estearum 47 minutes ago [-]
> If something claims to be conscious and we can't prove it's not, we have no choice but to believe it.
This is not a good test.
A dog won't claim to be conscious but clearly is, despite you not being able to prove one way or the other.
GPT-3 will claim to be conscious and (probably) isn't, despite you not being able to prove one way or the other.
sva_ 38 minutes ago [-]
> Edit: The average human tested scores 60%. So the machines are already smarter on an individual basis than the average human.
I think being better at this particular benchmark does not imply they're 'smarter'.
woah 50 minutes ago [-]
> If something claims to be conscious and we can't prove it's not, we have no choice but to believe it.
Can you "prove" that GPT2 isn't concious?
mapontosevenths 12 minutes ago [-]
If we equate self awareness with consciousness then yes. Several papers have now shown that SOTA models have self awareness of at least a limited sort. [0][1]
As far as I'm aware no one has ever proven that for GPT 2, but the methodology for testing it is available if you're interested.
I don't think the creator believes ARC3 can't be solved but rather that it can't be solved "efficiently" and >$13 per task for ARC2 is certainly not efficient.
But at this rate, the people who talk about the goal posts shifting even once we achieve AGI may end up correct, though I don't think this benchmark is particularly great either.
fishpham 3 hours ago [-]
Yes, but benchmarks like this are often flawed because leading model labs frequently participate in 'benchmarkmaxxing' - ie improvements on ARC-AGI2 don't necessarily indicate similar improvements in other areas (though it does seem like this is a step function increase in intelligence for the Gemini line of models)
layer8 3 hours ago [-]
Isn’t the point of ARC that you can’t train against it? Or doesn’t it achieve that goal anymore somehow?
egeozcan 2 hours ago [-]
How can you make sure of that? AFAIK, these SOTA models run exclusively on their developers hardware. So any test, any benchmark, anything you do, does leak per definition. Considering the nature of us humans and the typical prisoners dilemma, I don't see how they wouldn't focus on improving benchmarks even when it gets a bit... shady?
I tell this as a person who really enjoys AI by the way.
WarmWash 29 minutes ago [-]
Because the gains from spending time improving the model overall outweigh the gains from spending time individually training on benchmarks.
The pelican benchmark is a good example, because it's been representative of models ability to generate SVGs, not just pelicans on bikes.
1 hours ago [-]
theywillnvrknw 3 hours ago [-]
* that you weren't supposed to be able to
2 hours ago [-]
jstummbillig 3 hours ago [-]
Could it also be that the models are just a lot better than a year ago?
bigbadfeline 48 minutes ago [-]
> Could it also be that the models are just a lot better than a year ago?
No, the proof is in the pudding.
After AI we're having higher prices, higher deficits and lower standard of living. Electricity, computers and everything else costs more. "Doing better" can only be justified by that real benchmark.
If Gemini 3 DT was better we would have falling prices of electricity and everything else at least until they get to pre-2019 levels.
ctoth 32 minutes ago [-]
> If Gemini 3 DT was better we would have falling prices of electricity and everything else at least
Man, I've seen some maintenance folks down on the field before working on them goalposts but I'm pretty sure this is the first time I saw aliens from another Universe literally teleport in, grab the goalposts, and teleport out.
WarmWash 10 minutes ago [-]
You might call me crazy, but at least in 2024, consumers spent ~1% less of their income on expenses than 2019[2], which suggests that 2024 is more affordable than 2019.
This is from the BLS consumer survey report released in dec[1]
I don't understand what you want to tell us with this image.
fragmede 2 hours ago [-]
they're accusing GGP of moving the goalposts.
olalonde 3 hours ago [-]
Would be cool to have a benchmark with actually unsolved math and science questions, although I suspect models are still quite a long way from that level.
2 hours ago [-]
verdverm 3 hours ago [-]
Here's a good thread over 1+ month, as each model comes out
tl;dr - Pekka says Arc-AGI-2 is now toast as a benchmark
Aperocky 3 hours ago [-]
If you look at the problem space it is easy to see why it's toast, maybe there's intelligence in there, but hardly general.
verdverm 3 hours ago [-]
the best way I've seen this describes is "spikey" intelligence, really good at some points, those make the spikes
humans are the same way, we all have a unique spike pattern, interests and talents
ai are effectively the same spikes across instances, if simplified. I could argue self driving vs chatbots vs world models vs game playing might constitute enough variation. I would not say the same of Gemini vs Claude vs ... (instances), that's where I see "spikey clones"
Aperocky 3 hours ago [-]
You can get more spiky with AIs, whereas with human brain we are more hard wired.
So maybe we are forced to be more balanced and general whereas AI don't have to.
verdverm 3 hours ago [-]
I suspect the non-spikey part is the more interesting comparison
Why is it so easy for me to open the car door, get in, close the door, buckle up. You can do this in the dark and without looking.
There are an infinite number of little things like this you think zero about, take near zero energy, yet which are extremely hard for Ai
mNovak 2 hours ago [-]
I'm excited for the big jump in ARC-AGI scores from recent models, but no one should think for a second this is some leap in "general intelligence".
I joke to myself that the G in ARC-AGI is "graphical". I think what's held back models on ARC-AGI is their terrible spatial reasoning, and I'm guessing that's what the recent models have cracked.
Looking forward to ARC-AGI 3, which focuses on trial and error and exploring a set of constraints via games.
causal 2 hours ago [-]
Agreed. I love the elegance of ARC, but it always felt like a gotcha to give spatial reasoning challenges to token generators- and the fact that the token generators are somehow beating it anyway really says something.
throw310822 2 hours ago [-]
The average ARC AGI 2 score for a single human is around 60%.
"100% of tasks have been solved by at least 2 humans (many by more) in under 2 attempts. The average test-taker score was 60%."
Worth keeping in mind that in this case the test takers were random members of the general public. The score of e.g. people with bachelor's degrees in science and engineering would be significantly higher.
throw310822 46 minutes ago [-]
Random members of the public = average human beings. I thought those were already classified as General Intelligences.
imiric 6 minutes ago [-]
What is the point of comparing performance of these tools to humans? Machines have been able to accomplish specific tasks better than humans since the industrial revolution. Yet we don't ascribe intelligence to a calculator.
None of these benchmarks prove these tools are intelligent, let alone generally intelligent. The hubris and grift are exhausting.
colordrops 2 hours ago [-]
Wouldn't you deal with spatial reasoning by giving it access to a tool that structures the space in a way it can understand or just is a sub-model that can do spatial reasoning? These "general" models would serve as the frontal cortex while other models do specialized work. What is missing?
causal 2 hours ago [-]
That's a bit like saying just give blind people cameras so they can see.
culi 1 hours ago [-]
Yes but with a significant (logarithmic) increase in cost per task. The ARC-AGI site is less misleading and shows how GPT and Claude are not actually far behind
What’s reasonable? It’s less than minimum hourly wage in some countries.
willis936 1 hours ago [-]
Burned in seconds.
igravious 3 hours ago [-]
That's not a long time in the grand scheme of things.
throwup238 3 hours ago [-]
Speak for yourself. Five years is a long time to wait for my plans of world domination.
amelius 2 hours ago [-]
Yes, you better hurry.
mnicky 4 hours ago [-]
Well, fair comparison would be with GPT-5.x Pro, which is the same class of a model as Gemini Deep Think.
1 hours ago [-]
saberience 3 hours ago [-]
Arc-AGI (and Arc-AGI-2) is the most overhyped benchmark around though.
It's completely misnamed. It should be called useless visual puzzle benchmark 2.
It's a visual puzzle, making it way easier for humans than for models trained on text firstly. Secondly, it's not really that obvious or easy for humans to solve themselves!
So the idea that if an AI can solve "Arc-AGI" or "Arc-AGI-2" it's super smart or even "AGI" is frankly ridiculous. It's a puzzle that means nothing basically, other than the models can now solve "Arc-AGI"
CuriouslyC 3 hours ago [-]
The puzzles are calibrated for human solve rates, but otherwise I agree.
saberience 3 hours ago [-]
My two elderly parents cannot solve Arc-AGI puzzles, but can manage to navigate the physical world, their house, garden, make meals, clean the house, use the TV, etc.
I would say they do have "general intelligence", so whatever Arc-AGI is "solving" it's definitely not "AGI"
hmmmmmmmmmmmmmm 2 hours ago [-]
You are confusing fluid intelligence with crystallised intelligence.
casey2 2 hours ago [-]
I think you are making that confusion. Any robotic system in the place of his parents would fail with a few hours.
There are more novel tasks in a day than ARC provides.
hmmmmmmmmmmmmmm 2 hours ago [-]
Children have great levels of fluid intelligence, that's how they are able to learn to quickly navigate in a world that they are still very new to. Seniors with decreasing capacity increasingly rely on crystallised intelligence, that's why they can still perform tasks like driving a car but can fail at completely novel tasks, sometimes even using a smartphone if they have not used one before.
zeroonetwothree 55 minutes ago [-]
It really depends on motivation. My 90 year old grandmother can use a smartphone just fine since she needs it to see pictures of her (great) grandkids.
karmasimida 4 hours ago [-]
It is over
baal80spam 4 hours ago [-]
I for one welcome our new AI overlords.
logicprog 2 hours ago [-]
Is it me or is the rate of model release is accelerating to an absurd degree? Today we have Gemini 3 Deep Think and GPT 5.3 Codex Spark. Yesterday we had GLM5 and MiniMax M2.5. Five days before that we had Opus 4.6 and GPT 5.3. Then maybe two weeks I think before that we had Kimi K2.5.
i5heu 2 hours ago [-]
I think it is because of the Chinese new year.
The Chinese labs like to publish their models arround the Chinese new year, and the US labs do not want to let a DeepSeek R1 (20 January 2025) impact event happen again, so i guess they publish models that are more capable then what they imagine Chinese labs are yet capable of producing.
woah 15 minutes ago [-]
Singularity or just Chinese New Year?
aliston 2 hours ago [-]
I'm having trouble just keeping track of all these different types of models.
Is "Gemini 3 Deep Think" even technically a model? From what I've gathered, it is built on top of Gemini 3 Pro, and appears to be adding specific thinking capabilities, more akin to adding subagents than a truly new foundational model like Opus 4.6.
Also, I don't understand the comments about Google being behind in agentic workflows. I know that the typical use of, say, Claude Code feels agentic, but also a lot of folks are using separate agent harnesses like OpenClaw anyway. You could just as easily plug Gemini 3 Pro into OpenClaw as you can Opus, right?
Can someone help me understand these distinctions? Very confused, especially regarding the agent terminology. Much appreciated!
logicprog 59 minutes ago [-]
> Also, I don't understand the comments about Google being behind in agentic workflows.
It has to do with how the model is RL'd. It's not that Gemini can't be used with various agentic harnesses, like open code or open claw or theoretically even claude code. It's just that the model is trained less effectively to work with those harnesses, so it produces worse results.
re-thc 2 hours ago [-]
There are hints this is a preview to Gemini 3.1.
brokencode 2 hours ago [-]
They are using the current models to help develop even smarter models. Each generation of model can help even more for the next generation.
I don’t think it’s hyperbolic to say that we may be only a single digit number of years away from the singularity.
lm28469 2 hours ago [-]
I must be holding these things wrong because I'm not seeing any of these God like superpowers everyone seem to enjoy.
brokencode 54 minutes ago [-]
Who said they’re godlike today?
And yes, you are probably using them wrong if you don’t find them useful or don’t see the rapid improvement.
lm28469 44 minutes ago [-]
Let's come back in 12 months and discuss your singularity then. Meanwhile I spent like $30 on a few models as a test yesterday, none of them could tell me why my goroutine system was failing, even though it was painfully obvious (I purposefully added one too many wg.Done), gemini, codex, minimax 2.5, they all shat the bed on a very obvious problem but I am to believe they're 98% conscious and better at logic and math than 99% of the population.
Every new model release neckbeards come out of the basements to tell us the singularity will be there in two more weeks
BeetleB 41 seconds ago [-]
On the flip side, twice I put about 800K tokens of code into Gemini and asked it to find why my code was misbehaving, and it found it.
This was Gemini 2.5 Pro. A whole generation old.
woah 14 minutes ago [-]
Post the file here
brokencode 34 minutes ago [-]
You are fighting straw men here. Any further discussion would be pointless.
Izikiel43 4 minutes ago [-]
Out of curiosity, did you give a test for them to validate the code?
I had a test failing because I introduced a silly comparison bug (> instead of <), and claude 4.6 opus figured out it wasn't the test the problem, but the code and fixed the bug (which I had missed).
logicprog 36 minutes ago [-]
Meanwhile I've been using Kimi K2T and K2.5 to work in Go with a fair amount of concurrency and it's been able to write concurrent Go code and debug issues with goroutines equal to, and much more complex then, your issue, involving race conditions and more, just fine.
(Note that org-lsp has a much improved version of the same indexer as oxen; the first was purely my design, the second I decided to listen to K2.5 more and it found a bunch of potential race conditions and fixed them)
shrug
rogerkirkness 2 hours ago [-]
Fast takeoff.
redox99 2 hours ago [-]
There's more compute now than before.
bpodgursky 2 hours ago [-]
Anthropic took the day off to do a $30B raise at a $380B valuation.
IhateAI 2 hours ago [-]
Most ridiculous valuation in the history of markets. Cant wait to watch these compsnies crash snd burn when people give up on the slot machine.
andxor 23 minutes ago [-]
As usual don't take financial advice from HN folks!
kgwgk 2 hours ago [-]
WeWork almost IPO’s at $50bn. It was also a nice crash and burn.
jascha_eng 1 hours ago [-]
Why? They had $10+ billion arr run rate in 2025 trippeled from 2024
I mean 30x is a lot but also not insane at that growth rate right?
xnx 4 hours ago [-]
Google is absolutely running away with it. The greatest trick they ever pulled was letting people think they were behind.
wiseowise 2 hours ago [-]
Their models might be impressive, but their products absolutely suck donkey balls. I’ve given Gemini web/cli two months and ran away back to ChatGPT. Seriously, it would just COMPLETELY forget context mid dialog. When asked about improving air quality it just gave me a list of (mediocre) air purifiers without asking for any context whatsoever, and I can list thousands of conversations like that. Shopping or comparing options is just nonexistent.
It uses Russian propaganda sources for answers and switches to Chinese mid sentence (!), while explaining some generic Python functionality.
It’s an embarrassment and I don’t know how they justify 20 euro price tag on it.
mavamaarten 1 hours ago [-]
I agree. On top of that, in true Google style, basic things just don't work.
Any time I upload an attachment, it just fails with something vague like "couldn't process file". Whether that's a simple .MD or .txt with less than 100 lines or a PDF. I tried making a gem today. It just wouldn't let me save it, with some vague error too.
I also tried having it read and write stuff to "my stuff" and Google drive. But it would consistently write but not be able to read from it again. Or would read one file from Google drive and ignore everything else.
Their models are seriously impressive. But as usual Google sucks at making them work well in real products.
davoneus 26 minutes ago [-]
I don't find that at all. At work, we've no access to the API, so we have to force feed a dozen (or more) documents, code and instruction prompts through the web interface upload interface. The only failures I've ever had in well over 300 sessions were due to connectivity issues, not interface failures.
Context window blowouts? All the time, but never document upload failures.
sequin 18 minutes ago [-]
How can the models be impressive if they switch to Chinese mid-sentence? I've observed those bizarre bugs too. Even GPT-3 didn't have those. Maybe GPT-2 did. It's actually impressive that they managed to botch it so badly.
Google is great at some things, but this isn't it.
chermanowicz 53 minutes ago [-]
It's so capable at some things, and others are garbage.
I uploaded a photo of some words for a spelling bee and asked it to quiz my kid on the words. The first word it asked, wasn't on the list. After multiple attempts to get it to start asking only the words in the uploaded pic, it did, and then would get the spellings wrong in the Q&A. I gave up.
kilroy123 45 minutes ago [-]
Sadly true.
It is also one of the worst models to have a sort of ongoing conversation with.
HardCodedBias 2 hours ago [-]
Their models are absolutely not impressive.
Not a single person is using it for coding (outside of Google itself).
Maybe some people on a very generous free plan.
Their model is a fine mid 2025 model, backed by enormous compute resources and an army of GDM engineers to help the “researchers” keep the model on task as it traverses the “tree of thoughts”.
But that isn’t “the model” that’s an old model backed by massive money.
Ozzie_osman 2 hours ago [-]
Peacetime Google is not like wartime Google.
Peacetime Google is slow, bumbling, bureaucratic. Wartime Google gets shit done.
nutjob2 2 hours ago [-]
OpenAI is the best thing that happened to Google apparently.
RationPhantoms 57 minutes ago [-]
Competition always is. I think there was a real fear that their core product was going to be replaced. They're already cannibalizing it internally so it was THE wake up call.
lern_too_spel 59 minutes ago [-]
Wartime Google gave us Google+. Wartime Google is still bumbling, and despite OpenAI's numerous missteps, I don't think it has to worry about Google hurting its business yet.
kenjackson 2 hours ago [-]
But wait two hours for what OpenAI has! I love the competition and how someone just a few days ago was telling how ARC-AGI-2 was proof that LLMs can't reason. The goalposts will shift again. I feel like most of human endeavor will soon be just about trying to continuously show that AI's don't have AGI.
kilpikaarna 1 hours ago [-]
> I feel like most of human endeavor will soon be just about trying to continuously show that AI's don't have AGI.
I think you overestimate how much your average person-on-the-street cares about LLM benchmarks. They already treat ChatGPT or whichever as generally intelligent (including to their own detriment), are frustrated about their social media feeds filling up with slop and, maybe, if they're white-collar, worry about their jobs disappearing due to AI. Apart from a tiny minority in some specific field, people already know themselves to be less intelligent along any measurable axis than someone somewhere.
7777332215 2 hours ago [-]
Soon they can drop the bioweapon to welcome our replacement.
nutjob2 2 hours ago [-]
"AGI" doesn't mean anything concrete, so it's all a bunch of non-sequiturs. Your goalposts don't exist.
Anyone with any sense is interested in how well these tools work and how they can be harnessed, not some imaginary milestone that is not defined and cannot be measured.
kenjackson 1 hours ago [-]
I agree. I think the emergence of LLMs have shown that AGI really has no teeth. I think for decades the Turing test was viewed as the gold standard, but it's clear that there doesn't appear to be any good metric.
amunozo 4 hours ago [-]
Those black nazis in the first image model were a cause of inside trading.
naasking 2 hours ago [-]
Google is still behind the largest models I'd say, in real world utility. Gemini 3 Pro still has many issues.
Razengan 2 hours ago [-]
Gemini's UX (and of course privacy cred as with anything Google) is the worst of all the AI apps. In the eyes of the Common Man, it's UI that will win out, and ChatGPT's is still the best.
alexpotato 2 hours ago [-]
> Gemini's UX ... is the worst of all the AI apps
Been using Gemini + OpenCode for the past couple weeks.
Suddenly, I get a "you need a Gemini Access Code license" error but when you go to the project page there is no mention of this or how to get the license.
You really feel the "We're the phone company and we don't care. Why? Because we don't have to." [0] when you use these Google products.
PS for those that don't get the reference: US phone companies in the 1970s had a monopoly on local and long distance phone service. Similar to Google for search/ads (really a "near" monopoly but close enough).
They don't even let you have multiple chats if you disable their "App Activity" or whatever (wtf is with that ass naming? they don't even have a "Privacy" section in their settings the last time I checked)
and when I swap back into the Gemini app on my iPhone after a minute or so the chat disappears. and other weird passive-aggressive take-my-toys-away behavior if you don't bare your body and soul to Googlezebub.
ChatGPT and Grok work so much better without accounts or with high privacy settings.
Requests regularly time out, the whole window freezes, it gets stuck in schizophrenic loops, edits cannot be reverted and more.
It doesn't even come close to Claude or ChatGPT.
jonathanstrange 2 hours ago [-]
You mean AI Studio or something like that, right? Because I can't see a problem with Google's standard chat interface. All other AI offerings are confusing both regarding their intended use and their UX, though, I have to concur with that.
ergonaught 2 hours ago [-]
The lack of "projects" alone makes their chat interface really unpleasant compared to ChatGPT and Claude.
xnx 2 hours ago [-]
AI Studio is also significantly improved as of yesterday.
wiseowise 2 hours ago [-]
No projects, completely forgets context mid dialog, mediocre responses even on thinking, research got kneecapped somehow and is completely uses now, uses propaganda Russian videos as the search material (what’s wrong with you, Google?), janky on mobile, consumes GIGABYTES of RAM on web (seriously, what the fuck?). Left a couple of tabs over night, Mac is almost complete frozen because 10 tabs consumed 8 GBs of RAM doing nothing. It’s a complete joke.
dfdsf2 4 hours ago [-]
Trick? Lol not a chance. Alphabet is a pure play tech firm that has to produce products to make the tech accessible. They really lack in the latter and this is visible when you see the interactions of their VP's. Luckily for them, if you start to create enough of a lead with the tech, you get many chances to sort out the product stuff.
dakolli 3 hours ago [-]
You sound like Russ Hanneman from SV
s-kymon 3 hours ago [-]
It's not about how much you earn. It's about what you're worth.
Scene_Cast2 2 hours ago [-]
It's a shame that it's not on OpenRouter. I hate platform lock-in, but the top-tier "deep think" models have been increasingly requiring the use of their own platform.
raybb 2 hours ago [-]
OpenRouter is pretty great but I think litellm does a very good job and it's not a platform middle man, just a python library. That being said, I have tried it with the deep think models.
The arc-agi-2 score (84.6%) is from the semi-private eval set. If gemini-3-deepthink gets above 85% on the private eval set, it will be considered "solved"
Interestingly, the title of that PDF calls it "Gemini 3.1 Pro". Guess that's dropping soon.
sigmar 4 hours ago [-]
I looked at the file name but not the document title (specifically because I was wondering if this is 3.1). Good spot.
edit: they just removed the reference to "3.1" from the pdf
josalhor 3 hours ago [-]
I think this is 3.1 (3.0 Pro with the RL improv of 3.0 Flash).
But they probably decided to market it as Deep Think because why not charge more for it.
WarmWash 2 hours ago [-]
The Deep Think moniker is for parallel compute models though, not long CoT like pro models.
It's possible though that deep think 3 is running 3.1 models under the hood.
staticman2 3 hours ago [-]
That's odd considering 3.0 is still labeled a "preview" release.
> If gemini-3-deepthink gets above 85% on the private eval set, it will be considered "solved"
They never will do on private set, because it would mean its being leaked to google.
Decabytes 30 minutes ago [-]
Gemini has always felt like someone who was book smart to me. It knows a lot of things. But if you ask it do anything that is offscript it completely falls apart
dwringer 11 minutes ago [-]
I strongly suspect there's a major component of this type of experience being that people develop a way of talking to a particular LLM that's very efficient and works well for them with it, but is in many respects non-transferable to rival models. For instance, in my experience, OpenAI models are remarkably worse than Google models in basically any criterion I could imagine; however, I've spent most of my time using the Google ones and it's only during this time that the differences became apparent and, over time, much more pronounced. I would not be surprised at all to learn that people who chose to primarily use Anthropic or OpenAI models during that time had an exactly analogous experience that convinced them their model was the best.
esafak 26 minutes ago [-]
I'd rather say it has a mind of its own; it does things its way. But I have not tested this model.
vkazanov 20 minutes ago [-]
Well, one thing i know for sure: it reliably misplaces parentheses in lisps.
esafak 14 minutes ago [-]
Clearly, the AI is trying to steer you towards the ML family of languages for its better type system, performance, and concurrency ;)
jetter 2 hours ago [-]
it is interesting that the video demo is generating .stl model.
I run a lot of tests of LLMs generating OpenSCAD code (as I have recently launched https://modelrift.com text-to-CAD AI editor) and Gemini 3 family LLMs are actually giving the best price-to-performance ratio now. But they are very, VERY far from being able to spit out a complex OpenSCAD model in one shot. So, I had to implement a full fledged "screenshot-vibe-coding" workflow where you draw arrows on 3d model snapshot to explain to LLM what is wrong with the geometry. Without human in the loop, all top tier LLMs hallucinate at debugging 3d geometry in agentic mode - and fail spectacularly.
mchusma 38 minutes ago [-]
Hey, my 9 year old son uses modelrift for creating things for his 3d printer, its great! Product feedback:
1. You should probably ask me to pay now, I feel like i've used it enough.
2. You need a main dashboard page with a history of sessions. He thought he lost a file and I had to dig in the billing history to get a UUID I thought was it and generate the url. I would say naming sessions is important, and could be done with small LLM after the users initial prompt.
3. I don't think I like the default 3d model in there once I have done something, blank would be better.
We download the stl and import to bambu. Works pretty well. A direct push would be nice, but not necessary.
gundmc 59 minutes ago [-]
Yes, I've been waiting for a real breakthrough with regard to 3D parametric models and I don't think think this is it. The proprietary nature of the major players (Creo, Solidworks, NX, etc) is a major drag. Sure there's STP, but there's too much design intent and feature loss there. I don't think OpenSCAD has the critical mass of mindshare or training data at this point, but maybe it's the best chance to force a change.
lern_too_spel 50 minutes ago [-]
If you want that to get better, you need to produce a 3d model benchmark and popularize it. You can start with a pelican riding a bicycle with working bicycle.
simianwords 4 hours ago [-]
OT but my intuition says that there’s a spectrum
- non thinking models
- thinking models
- best of N models like deep think an gpt pro
Each one is of a certain computational complexity. Simplifying a bit, I think they map to - linear, quadratic and n^3 respectively.
I think there are certain class of problems that can’t be solved without thinking because it necessarily involves writing in a scratchpad. And same for best of N which involves exploring.
Two open questions
1) what’s the higher level here, is there a 4th option?
2) can a sufficiently large non thinking model perform the same as a smaller thinking?
futureshock 53 minutes ago [-]
I think step 4 is the agent swarm. Manager model gets the prompt and spins up a swarm of looping subagents, maybe assigns them different approaches or subtasks, then reviews results, refines the context files and redeploys the swarm on a loop till the problem is solved or your credit card is declined.
simianwords 47 minutes ago [-]
i think this is the right answer
NitpickLawyer 4 hours ago [-]
> best of N models like deep think an gpt pro
Yeah, these are made possible largely by better use at high context lengths. You also need a step that gathers all the Ns and selects the best ideas / parts and compiles the final output. Goog have been SotA at useful long context for a while now (since 2.5 I'd say). Many others have come with "1M context", but their usefulness after 100k-200k is iffy.
What's even more interesting than maj@n or best of n is pass@n. For a lot of applications youc an frame the question and search space such that pass@n is your success rate. Think security exploit finding. Or optimisation problems with quick checks (better algos, kernels, infra routing, etc). It doesn't matter how good your pass@1 or avg@n is, all you care is that you find more as you spend more time. Literally throwing money at the problem.
mnicky 4 hours ago [-]
> can a sufficiently large non thinking model perform the same as a smaller thinking?
Models from Anthropic have always been excellent at this. See e.g. https://imgur.com/a/EwW9H6q (top-left Opus 4.6 is without thinking).
simianwords 4 hours ago [-]
its interesting that opus 4.6 added a paramter to make it think extra hard.
Legend2440 12 minutes ago [-]
I'm really interested in the 3D STL-from-photo process they demo in the video.
Not interested enough to pay $250 to try it out though.
aliljet 1 hours ago [-]
The problem here is that it looks like this is released with almost no real access. How are people using this without submitting to a $250/mo subscription?
andxor 21 minutes ago [-]
People are paying for the subscriptions.
Metacelsus 4 hours ago [-]
According to benchmarks in the announcement, healthily ahead of Claude 4.6. I guess they didn't test ChatGPT 5.3 though.
Google has definitely been pulling ahead in AI over the last few months. I've been using Gemini and finding it's better than the other models (especially for biology where it doesn't refuse to answer harmless questions).
CuriouslyC 3 hours ago [-]
Google is way ahead in visual AI and world modelling. They're lagging hard in agentic AI and autonomous behavior.
scarmig 57 minutes ago [-]
> especially for biology where it doesn't refuse to answer harmless questions
Usually, when you decrease false positive rates, you increase false negative rates.
Maybe this doesn't matter for models at their current capabilities, but if you believe that AGI is imminent, a bit of conservatism seems responsible.
throwup238 4 hours ago [-]
The general purpose ChatGpt 5.3 hasn’t been released yet, just 5.3-codex.
neilellis 4 hours ago [-]
It's ahead in raw power but not in function. Like it's got the worlds fast engine but one gear! Trouble is some benchmarks only measure horse power.
NitpickLawyer 4 hours ago [-]
> Trouble is some benchmarks only measure horse power.
IMO it's the other way around. Benchmarks only measure applied horse power on a set plane, with no friction and your elephant is a point sphere. Goog's models have always punched over what benchmarks said, in real world use @ high context. They don't focus on "agentic this" or "specialised that", but the raw models, with good guidance are workhorses. I don't know any other models where you can throw lots of docs at it and get proper context following and data extraction from wherever it's at to where you'd need it.
Davidzheng 4 hours ago [-]
I gather that 4.6 strengths are in long context agentic workflows? At least over Gemini 3 pro preview, opus 4.6 seems to have a lot of advantages
verdverm 3 hours ago [-]
It's a giant game of leapfrog, shift or stretch time out a bit and they all look equivalent
nkzd 3 hours ago [-]
Google models and CLI harness feels behind in agentic coding compared OpenAI and Antrophic
simianwords 4 hours ago [-]
The comparison should be with GPT 5.2 pro which has been used successfully to solve open math problems.
siva7 3 hours ago [-]
I can't shake of the feeling that Googles Deep Think Models are not really different models but just the old ones being run with higher number of parallel subagents, something you can do by yourself with their base model and opencode.
Davidzheng 3 hours ago [-]
And after i do that, how do i combine the output of 1000 subagents into one output? (Im not being snarky here, i think it's a nontrivial problem)
tifik 2 hours ago [-]
The idea is that each subagent is focused on a specific part of the problem and can use its entire context window for a more focused subtask than the overall one. So ideally the results arent conflicting, they are complimentary. And you just have a system that merges them.. likely another agent.
mattlondon 2 hours ago [-]
You just pipe it to another agent to do the reduce step (i.e. fan-in) of the mapreduce (fan-out)
It's agents all the way down.
jonathanstrange 2 hours ago [-]
Start with 1024 and use half the number of agents each turn to distill the final result.
sinuhe69 3 hours ago [-]
I'm pretty certain that DeepMind (and all other labs) will try their frontier (and even private) models on First Proof [1].
And I wonder how Gemini Deep Think will fare. My guess is that it will get half the way on some problems. But we will have to take an absence as a failure, because nobody wants to publish a negative result, even though it's so important for scientific research.
The 1st proof original solutions are due to be published in about 24h, AIUI.
2 hours ago [-]
neilellis 4 hours ago [-]
Less than a year to destroy Arc-AGI-2 - wow.
Davidzheng 4 hours ago [-]
I unironically believe that arc-agi-3 will have a introduction to solved time of 1 month
ACCount37 46 minutes ago [-]
Not very likely?
ARC-AGI-3 has a nasty combo of spatial reasoning + explore/exploit. It's basically adversarial vs current AIs.
etyhhgfff 3 hours ago [-]
The AGI bar has to be set even higher, yet again.
dakolli 2 hours ago [-]
wow solving useless puzzles, such a useful metric!
modeless 2 hours ago [-]
It's still useful as a benchmark of cost/efficiency.
XCSme 3 hours ago [-]
But why only a +0.5% increase for MMMU-Pro?
kingstnap 1 hours ago [-]
Its possibly label noise. But you can't tell from a single number.
You would need to check to see if everyone is having mistakes on the same 20% or different 20%. If its the same 20% either those questions are really hard, or they are keyed incorrectly, or they aren't stated with enough context to actually solve the problem.
It happens. Old MMLU non pro had a lot of wrong answers. Simple things like MNIST have digits labeled incorrect or drawn so badly its not even a digit anymore.
kenjackson 2 hours ago [-]
Everyone is already at 80% for that one. Crazy that we were just at 50% with GPT-4o not that long ago.
saberience 3 hours ago [-]
It's a useless meaningless benchmark though, it just got a catchy name, as in, if the models solve this it means they have "AGI", which is clearly rubbish.
Arc-AGI score isn't correlated with anything useful.
Legend2440 15 minutes ago [-]
It's correlated with the ability to solve logic puzzles.
It's also interesting because it's very very hard for base LLMs, even if you try to "cheat" by training on millions of ARC-like problems. Reasoning LLMs show genuine improvement on this type of problem.
jabedude 3 hours ago [-]
how would we actually objectively measure a model to see if it is AGI if not with benchmarks like arc-AGI?
WarmWash 1 hours ago [-]
Give it a prompt like
>can u make the progm for helps that with what in need for shpping good cheap products that will display them on screen and have me let the best one to get so that i can quickly hav it at home
And get back an automatic coupon code app like the user actually wanted.
37 minutes ago [-]
Dirak 1 hours ago [-]
Praying this isn't another Llama4 situation where the benchmark numbers are cooked. 84.6% on Arc-AGI is incredible!
ramshanker 3 hours ago [-]
Do we get any model architecture details like parameter size etc.? Few months back, we used to talk more on this, now it's mostly about model capabilities.
Davidzheng 3 hours ago [-]
I'm honestly not sure what you mean? The frontier labs have kept arch as secrets since gpt3.5
willis936 1 hours ago [-]
At the very least gemini 3's flyer claims 1T parameters.
vessenes 4 hours ago [-]
Not trained for agentic workflows yet unfortunately - this looks like it will be fantastic when they have an agent friendly one. Super exciting.
dakolli 3 hours ago [-]
Its really weird how you all are begging to be replaced by llms, you think if agentic workflows get good enough you're going to keep your job? Or not have your salary reduced by 50%?
If Agents get good enough it's not going to build some profitable startup for you (or whatever people think they're doing with the llm slot machines) because that implies that anyone else with access to that agent can just copy you, its what they're designed to do... launder IP/Copyright. Its weird to see people get excited for this technology.
None of this good. We are simply going to have our workforces replaced by assets owned by Google, Anthropic and OpenAI. We'll all be fighting for the same barista jobs, or miserable factory jobs. Take note on how all these CEOs are trying to make it sound cool to "go to trade school" or how we need "strong American workers to work in factories".
sgillen 39 minutes ago [-]
I think a lot of people assume they will become highly paid Agent orchestrators or some such. I don't think anyone really knows where things are heading.
ergonaught 2 hours ago [-]
Most folks don't seem to think that far down the line, or they haven't caught on to the reality that the people who actually make decisions will make the obvious kind of decisions (ex: fire the humans, cut the pay, etc) that they already make.
newswasboring 20 minutes ago [-]
You don't hate AI, you hate capitalism. All the problems you have listed are not AI issues, its this crappy system where efficiency gains always end up with the capital owners.
OtomotO 2 hours ago [-]
Or we just end capitalism.
French revolution style.
shrugs
uxhoiuewfhhiu 11 minutes ago [-]
Let's start with you.
dakolli 2 hours ago [-]
Well I honestly think this is the solution. It's much harder to do French Revolution V2 though if they've used ML to perfect people's recommendation algorithms to psyop them into fighting wars on behalf of capitalists.
I imagine llm job automation will make people so poor that they beg to fight in wars, and instead of turning that energy against he people who created the problem they'll be met with hours of psyops that direct that energy to Chinese people or whatever.
We will see.
3 hours ago [-]
simonw 3 hours ago [-]
The pelican riding a bicycle is excellent. I think it's the best I've seen.
I routinely check out the pelicans you post and I do agree, this is the best yet. It seemed to me that the wings/arms were such a big hangup for these generators.
Manabu-eo 3 hours ago [-]
How likely this problem is already on the training set by now?
Why would they train on that? Why not just hire someone to make a few examples.
simonw 2 hours ago [-]
I look forward to them trying. I'll know when the pelican riding a bicycle is good but the ocelot riding a skateboard sucks.
suddenlybananas 1 hours ago [-]
But they could just train on an assortment of animals and vehicles. It's the kind of relatively narrow domain where NNs could reasonably interpolate.
simonw 1 hours ago [-]
The idea that an AI lab would pay a small army of human artists to create training data for $animal on $transport just to cheat on my stupid benchmark delights me.
suddenlybananas 1 hours ago [-]
When you're spending trillions on capex, paying a couple of people to make some doodles in SVGs would not be a big expense.
simonw 45 minutes ago [-]
The embarrassment of getting caught doing that would be expensive.
throwup238 3 hours ago [-]
For every combination of animal and vehicle? Very unlikely.
The beauty of this benchmark is that it takes all of two seconds to come up with your own unique one. A seahorse on a unicycle. A platypus flying a glider. A man’o’war piloting a Portuguese man of war. Whatever you want.
recursive 3 hours ago [-]
No, not every combination. The question is about the specific combination of a pelican on a bicycle. It might be easy to come up with another test, but we're looking at the results from a particular one here.
svara 2 hours ago [-]
More likely you would just train for emitting svg for some description of a scene and create training data from raster images.
zarzavat 3 hours ago [-]
You can always ask for a tyrannosaurus driving a tank.
verdverm 3 hours ago [-]
I've heard it posited that the reason the frontier companies are frontier is because they have custom data and evals. This is what I would do too
3 hours ago [-]
enraged_camel 2 hours ago [-]
Is there a list of these for each model, that you've catalogued somewhere?
throwup238 3 hours ago [-]
The reflection of the sun in the water is completely wrong. LLMs are still useless. (/s)
margalabargala 2 hours ago [-]
It's not actually, look up some photos of the sun setting over the ocean. Here's an example:
It's worth noting that you mean excellent in terms of prior AI output. I'm pretty sure this wouldn't be considered excellent from a "human made art" perspective. In other words, it's still got a ways to go!
Edit: someone needs to explain why this comment is getting downvoted, because I don't understand. Did someone's ego get hurt, or what?
gs17 3 hours ago [-]
It depends, if you meant from a human coding an SVG "manually" the same way, I'd still say this is excellent (minus the reflection issue). If you meant a human using a proper vector editor, then yeah.
fvdessen 2 hours ago [-]
maybe you're a pro vector artist but I couldn't create such a cool one myself in illustrator tbh
dfdsf2 3 hours ago [-]
Indeed. And when you factor in the amount invested... yeah it looks less impressive. The question is how much more money needs to be invested to get this thing closer to reality? And not just in this instance. But for any instance e.g. a seahorse on a bike.
saberience 3 hours ago [-]
Do you have to still keep trying to bang on about this relentlessly?
It was sort of humorous for the maybe first 2 iterations, now it's tacky, cheesy, and just relentless self-promotion.
Again, like I said before, it's also a terrible benchmark.
jeanloolz 18 minutes ago [-]
I'll agree to disagree. In any thread about a new model, I personally expect the pelican comment to be out there. It's informative, ritualistic and frankly fun. Your comment however, is a little harsh. Why mad?
simonw 2 hours ago [-]
It being a terrible benchmark is the bit.
Davidzheng 3 hours ago [-]
Eh, i find it more of a not very informative but lighthearted commentary
dfdsf2 3 hours ago [-]
Highly disagree.
I was expecting something more realistic... the true test of what you are doing is how representative is the thing in relation to the real world. E.g. does the pelican look like a pelican as it exists in reality? This cartoon stuff is cute but doesnt pass muster in my view.
If it doesn't relate to the real world, then it most likely will have no real effect on the real economy. Pure and simple.
chriswarbo 2 hours ago [-]
I disagree. The task asks for an SVG; which is a vector format associated with line drawings, clipart and cartoons. I think it's good that models are picking up on that context.
In contrast, the only "realistic" SVGs I've seen are created using tools like potrace, and look terrible.
I also think the prompt itself, of a pelican on bicycle, is unrealistic and cartoonish; so making a cartoon is a good way to solve the task.
peaseagee 3 hours ago [-]
The request is for an SVG, generally _not_ the format for photorealistic images. If you want to start your own benchmark, feel free to ask for a photorealistic JPEG or PNG of a pelican riding a bicycle. Could be interesting to compare and contrast, honestly.
jonathanstrange 4 hours ago [-]
Unfortunately, it's only available in the Ultra subscription if it's available at all.
ismailmaj 3 hours ago [-]
top 10 elo in codeforces is pretty absurd
andrewstuart 3 hours ago [-]
Gemini was awesome and now it’s garbage.
It’s impossible for it to do anything but cut code down, drop features, lose stuff and give you less than the code you put in.
It’s puzzling because it spent months at the head of the pack now I don’t use it at all because why do I want any of those things when I’m doing development.
I’m a paid subscriber but there’s no point any more I’ll spend the money on Claude 4.6 instead.
ergonaught 2 hours ago [-]
It seems to be adept at reviewing/editing/critiquing, at least for my use cases. It always has something valuable to contribute from that perspective, but has been comparatively useless otherwise (outside of moats like "exclusive access to things involving YouTube").
halapro 2 hours ago [-]
I never found it useful for code. It produced garbage littered with gigantic comments.
Me: Remove comments
Literally Gemini: // Comments were removed
andrewstuart 2 hours ago [-]
It would make more sense to me if it had never been awesome.
mortsnort 23 minutes ago [-]
They may quantize the models after release to save money.
m3kw9 3 hours ago [-]
Gemini 3 Pro/Flash is stuck in preview for months now. Google is slow but they progress like a massive rock giant.
okokwhatever 3 hours ago [-]
I need to test the sketch creation a s a p. I need this in my life because learning to use Freecad is too difficult for a busy person like me (and frankly, also quite lazy)
sho_hn 3 hours ago [-]
FWIW, the FreeCAD 1.1 nightlies are much easier and more intuitive to use due to the addition of many on-canvas gizmos.
HardCodedBias 2 hours ago [-]
Always the same with Google.
Gemini has been way behind from the start.
They use the firehose of money from search to make it as close to free as possible so that they have some adoption numbers.
They use the firehose from search to pay for tons of researchers to hand hold academics so that their non-economic models and non-economic test-time-compute can solve isolated problems.
It's all so tiresome.
Try making models that are actually competitive, Google.
Sell them on the actual market and win on actual work product in millions of people lives.
@dang will often replace the post url & merge comments
HN guidelines prefer the original source over social posts linking to it.
aavci 3 hours ago [-]
Agreed - blog post is more appropriate than a twitter post
bschmidt720 2 hours ago [-]
[dead]
dperhar 3 hours ago [-]
Does anyone actually use Gemini 3 now? I cant stand its sleek salesy way of introduction, and it doesnt hold to instructions hard – makes it unapplicable for MECE breakdowns or for writing.
copperx 2 hours ago [-]
I do. It's excellent when paired with an MCP like context7.
throwa356262 3 hours ago [-]
I dont agree, Gemini 3 is pretty good, even the Lite version.
dperhar 2 hours ago [-]
What do you use it for and why? Genuinely curious
IhateAI 2 hours ago [-]
[dead]
jeffbee 2 hours ago [-]
It indeed departs from instructions pretty regularly. But I find it very useful and for the price it beats the world.
"The price" is the marginal price I am paying on top of my existing Google 1, YouTube Premium, and Google Fi subs, so basically nothing on the margin.
wetpaws 2 hours ago [-]
[dead]
Rendered at 21:26:27 GMT+0000 (Coordinated Universal Time) with Vercel.
Wow.
https://blog.google/innovation-and-ai/models-and-research/ge...
1. It's an LLM, not something trained to play Balatro specifically
2. Most (probably >99.9%) players can't do that at the first attempt
3. I don't think there are many people who posted their Balatro playthroughs in text form online
I think it's a much stronger signal of its 'generalness' than ARC-AGI. By the way, Deepseek can't play Balatro at all.
[0]: https://balatrobench.com/
Nonetheless I still think it's impressive that we have LLMs that can just do this now.
Eh, both myself and my partner did this. To be fair, we weren’t going in completely blind, and my partner hit a Legendary joker, but I think you might be slightly overstating the difficulty. I’m still impressed that Gemini did it.
There are *tons* of balatro content on YouTube though, and it makes absolutely zero doubt that Google is using YouTube content to train their model.
I really doubt it's playing completely blind
I ask because I cannot distinguish all the benchmarks by heart.
His definition of reaching AGI, as I understand it, is when it becomes impossible to construct the next version of ARC-AGI because we can no longer find tasks that are feasible for normal humans but unsolved by AI.
That is the best definition I've yet to read. If something claims to be conscious and we can't prove it's not, we have no choice but to believe it.
Thats said, I'm reminded of the impossible voting tests they used to give black people to prevent them from voting. We dont ask nearly so much proof from a human, we take their word for it. On the few occasions we did ask for proof it inevitably led to horrific abuse.
Edit: The average human tested scores 60%. So the machines are already smarter on an individual basis than the average human.
"Answer "I don't know" if you don't know an answer to one of the questions"
This is not a good test.
A dog won't claim to be conscious but clearly is, despite you not being able to prove one way or the other.
GPT-3 will claim to be conscious and (probably) isn't, despite you not being able to prove one way or the other.
I think being better at this particular benchmark does not imply they're 'smarter'.
Can you "prove" that GPT2 isn't concious?
As far as I'm aware no one has ever proven that for GPT 2, but the methodology for testing it is available if you're interested.
[0]https://arxiv.org/pdf/2501.11120
[1]https://transformer-circuits.pub/2025/introspection/index.ht...
But at this rate, the people who talk about the goal posts shifting even once we achieve AGI may end up correct, though I don't think this benchmark is particularly great either.
I tell this as a person who really enjoys AI by the way.
The pelican benchmark is a good example, because it's been representative of models ability to generate SVGs, not just pelicans on bikes.
No, the proof is in the pudding.
After AI we're having higher prices, higher deficits and lower standard of living. Electricity, computers and everything else costs more. "Doing better" can only be justified by that real benchmark.
If Gemini 3 DT was better we would have falling prices of electricity and everything else at least until they get to pre-2019 levels.
Man, I've seen some maintenance folks down on the field before working on them goalposts but I'm pretty sure this is the first time I saw aliens from another Universe literally teleport in, grab the goalposts, and teleport out.
This is from the BLS consumer survey report released in dec[1]
[1]https://www.bls.gov/news.release/cesan.nr0.htm
[2]https://www.bls.gov/opub/reports/consumer-expenditures/2019/
Prices are never going back to 2019 numbers though
https://bsky.app/profile/pekka.bsky.social/post/3meokmizvt22...
tl;dr - Pekka says Arc-AGI-2 is now toast as a benchmark
humans are the same way, we all have a unique spike pattern, interests and talents
ai are effectively the same spikes across instances, if simplified. I could argue self driving vs chatbots vs world models vs game playing might constitute enough variation. I would not say the same of Gemini vs Claude vs ... (instances), that's where I see "spikey clones"
So maybe we are forced to be more balanced and general whereas AI don't have to.
Why is it so easy for me to open the car door, get in, close the door, buckle up. You can do this in the dark and without looking.
There are an infinite number of little things like this you think zero about, take near zero energy, yet which are extremely hard for Ai
I joke to myself that the G in ARC-AGI is "graphical". I think what's held back models on ARC-AGI is their terrible spatial reasoning, and I'm guessing that's what the recent models have cracked.
Looking forward to ARC-AGI 3, which focuses on trial and error and exploring a set of constraints via games.
"100% of tasks have been solved by at least 2 humans (many by more) in under 2 attempts. The average test-taker score was 60%."
https://arcprize.org/arc-agi/2/
None of these benchmarks prove these tools are intelligent, let alone generally intelligent. The hubris and grift are exhausting.
https://arcprize.org/leaderboard
$13.62 per task - so we need another 5-10 years for the price to run this to become reasonable?
But the real question is if they just fit the model to the benchmark.
At current rates, price per equivalent output is dropping at 99.9% over 5 years.
That's basically $0.01 in 5 years.
Does it really need to be that cheap to be worth it?
Keep in mind, $0.01 in 5 years is worth less than $0.01 today.
It's completely misnamed. It should be called useless visual puzzle benchmark 2.
It's a visual puzzle, making it way easier for humans than for models trained on text firstly. Secondly, it's not really that obvious or easy for humans to solve themselves!
So the idea that if an AI can solve "Arc-AGI" or "Arc-AGI-2" it's super smart or even "AGI" is frankly ridiculous. It's a puzzle that means nothing basically, other than the models can now solve "Arc-AGI"
I would say they do have "general intelligence", so whatever Arc-AGI is "solving" it's definitely not "AGI"
There are more novel tasks in a day than ARC provides.
Is "Gemini 3 Deep Think" even technically a model? From what I've gathered, it is built on top of Gemini 3 Pro, and appears to be adding specific thinking capabilities, more akin to adding subagents than a truly new foundational model like Opus 4.6.
Also, I don't understand the comments about Google being behind in agentic workflows. I know that the typical use of, say, Claude Code feels agentic, but also a lot of folks are using separate agent harnesses like OpenClaw anyway. You could just as easily plug Gemini 3 Pro into OpenClaw as you can Opus, right?
Can someone help me understand these distinctions? Very confused, especially regarding the agent terminology. Much appreciated!
It has to do with how the model is RL'd. It's not that Gemini can't be used with various agentic harnesses, like open code or open claw or theoretically even claude code. It's just that the model is trained less effectively to work with those harnesses, so it produces worse results.
I don’t think it’s hyperbolic to say that we may be only a single digit number of years away from the singularity.
And yes, you are probably using them wrong if you don’t find them useful or don’t see the rapid improvement.
Every new model release neckbeards come out of the basements to tell us the singularity will be there in two more weeks
This was Gemini 2.5 Pro. A whole generation old.
I had a test failing because I introduced a silly comparison bug (> instead of <), and claude 4.6 opus figured out it wasn't the test the problem, but the code and fixed the bug (which I had missed).
Projects:
https://github.com/alexispurslane/oxen
https://github.com/alexispurslane/org-lsp
(Note that org-lsp has a much improved version of the same indexer as oxen; the first was purely my design, the second I decided to listen to K2.5 more and it found a bunch of potential race conditions and fixed them)
shrug
Any time I upload an attachment, it just fails with something vague like "couldn't process file". Whether that's a simple .MD or .txt with less than 100 lines or a PDF. I tried making a gem today. It just wouldn't let me save it, with some vague error too.
I also tried having it read and write stuff to "my stuff" and Google drive. But it would consistently write but not be able to read from it again. Or would read one file from Google drive and ignore everything else.
Their models are seriously impressive. But as usual Google sucks at making them work well in real products.
Context window blowouts? All the time, but never document upload failures.
Google is great at some things, but this isn't it.
It is also one of the worst models to have a sort of ongoing conversation with.
Not a single person is using it for coding (outside of Google itself).
Maybe some people on a very generous free plan.
Their model is a fine mid 2025 model, backed by enormous compute resources and an army of GDM engineers to help the “researchers” keep the model on task as it traverses the “tree of thoughts”.
But that isn’t “the model” that’s an old model backed by massive money.
Peacetime Google is slow, bumbling, bureaucratic. Wartime Google gets shit done.
I think you overestimate how much your average person-on-the-street cares about LLM benchmarks. They already treat ChatGPT or whichever as generally intelligent (including to their own detriment), are frustrated about their social media feeds filling up with slop and, maybe, if they're white-collar, worry about their jobs disappearing due to AI. Apart from a tiny minority in some specific field, people already know themselves to be less intelligent along any measurable axis than someone somewhere.
Anyone with any sense is interested in how well these tools work and how they can be harnessed, not some imaginary milestone that is not defined and cannot be measured.
Been using Gemini + OpenCode for the past couple weeks.
Suddenly, I get a "you need a Gemini Access Code license" error but when you go to the project page there is no mention of this or how to get the license.
You really feel the "We're the phone company and we don't care. Why? Because we don't have to." [0] when you use these Google products.
PS for those that don't get the reference: US phone companies in the 1970s had a monopoly on local and long distance phone service. Similar to Google for search/ads (really a "near" monopoly but close enough).
0 - https://vimeo.com/355556831
Afaik, Google has had no breaches ever.
and when I swap back into the Gemini app on my iPhone after a minute or so the chat disappears. and other weird passive-aggressive take-my-toys-away behavior if you don't bare your body and soul to Googlezebub.
ChatGPT and Grok work so much better without accounts or with high privacy settings.
Requests regularly time out, the whole window freezes, it gets stuck in schizophrenic loops, edits cannot be reverted and more.
It doesn't even come close to Claude or ChatGPT.
https://docs.litellm.ai/docs/
The arc-agi-2 score (84.6%) is from the semi-private eval set. If gemini-3-deepthink gets above 85% on the private eval set, it will be considered "solved"
>Submit a solution which scores 85% on the ARC-AGI-2 private evaluation set and win $700K. https://arcprize.org/guide#overview
edit: they just removed the reference to "3.1" from the pdf
It's possible though that deep think 3 is running 3.1 models under the hood.
They never will do on private set, because it would mean its being leaked to google.
We download the stl and import to bambu. Works pretty well. A direct push would be nice, but not necessary.
- non thinking models
- thinking models
- best of N models like deep think an gpt pro
Each one is of a certain computational complexity. Simplifying a bit, I think they map to - linear, quadratic and n^3 respectively.
I think there are certain class of problems that can’t be solved without thinking because it necessarily involves writing in a scratchpad. And same for best of N which involves exploring.
Two open questions
1) what’s the higher level here, is there a 4th option?
2) can a sufficiently large non thinking model perform the same as a smaller thinking?
Yeah, these are made possible largely by better use at high context lengths. You also need a step that gathers all the Ns and selects the best ideas / parts and compiles the final output. Goog have been SotA at useful long context for a while now (since 2.5 I'd say). Many others have come with "1M context", but their usefulness after 100k-200k is iffy.
What's even more interesting than maj@n or best of n is pass@n. For a lot of applications youc an frame the question and search space such that pass@n is your success rate. Think security exploit finding. Or optimisation problems with quick checks (better algos, kernels, infra routing, etc). It doesn't matter how good your pass@1 or avg@n is, all you care is that you find more as you spend more time. Literally throwing money at the problem.
Models from Anthropic have always been excellent at this. See e.g. https://imgur.com/a/EwW9H6q (top-left Opus 4.6 is without thinking).
Not interested enough to pay $250 to try it out though.
Google has definitely been pulling ahead in AI over the last few months. I've been using Gemini and finding it's better than the other models (especially for biology where it doesn't refuse to answer harmless questions).
Usually, when you decrease false positive rates, you increase false negative rates.
Maybe this doesn't matter for models at their current capabilities, but if you believe that AGI is imminent, a bit of conservatism seems responsible.
IMO it's the other way around. Benchmarks only measure applied horse power on a set plane, with no friction and your elephant is a point sphere. Goog's models have always punched over what benchmarks said, in real world use @ high context. They don't focus on "agentic this" or "specialised that", but the raw models, with good guidance are workhorses. I don't know any other models where you can throw lots of docs at it and get proper context following and data extraction from wherever it's at to where you'd need it.
It's agents all the way down.
And I wonder how Gemini Deep Think will fare. My guess is that it will get half the way on some problems. But we will have to take an absence as a failure, because nobody wants to publish a negative result, even though it's so important for scientific research.
[1] https://1stproof.org/
ARC-AGI-3 has a nasty combo of spatial reasoning + explore/exploit. It's basically adversarial vs current AIs.
You would need to check to see if everyone is having mistakes on the same 20% or different 20%. If its the same 20% either those questions are really hard, or they are keyed incorrectly, or they aren't stated with enough context to actually solve the problem.
It happens. Old MMLU non pro had a lot of wrong answers. Simple things like MNIST have digits labeled incorrect or drawn so badly its not even a digit anymore.
Arc-AGI score isn't correlated with anything useful.
It's also interesting because it's very very hard for base LLMs, even if you try to "cheat" by training on millions of ARC-like problems. Reasoning LLMs show genuine improvement on this type of problem.
>can u make the progm for helps that with what in need for shpping good cheap products that will display them on screen and have me let the best one to get so that i can quickly hav it at home
And get back an automatic coupon code app like the user actually wanted.
If Agents get good enough it's not going to build some profitable startup for you (or whatever people think they're doing with the llm slot machines) because that implies that anyone else with access to that agent can just copy you, its what they're designed to do... launder IP/Copyright. Its weird to see people get excited for this technology.
None of this good. We are simply going to have our workforces replaced by assets owned by Google, Anthropic and OpenAI. We'll all be fighting for the same barista jobs, or miserable factory jobs. Take note on how all these CEOs are trying to make it sound cool to "go to trade school" or how we need "strong American workers to work in factories".
French revolution style.
shrugs
I imagine llm job automation will make people so poor that they beg to fight in wars, and instead of turning that energy against he people who created the problem they'll be met with hours of psyops that direct that energy to Chinese people or whatever.
We will see.
https://simonwillison.net/2026/Feb/12/gemini-3-deep-think/
The beauty of this benchmark is that it takes all of two seconds to come up with your own unique one. A seahorse on a unicycle. A platypus flying a glider. A man’o’war piloting a Portuguese man of war. Whatever you want.
https://stockcake.com/i/sunset-over-ocean_1317824_81961
https://stockcake.com/i/serene-ocean-sunset_1152191_440307
Edit: someone needs to explain why this comment is getting downvoted, because I don't understand. Did someone's ego get hurt, or what?
It was sort of humorous for the maybe first 2 iterations, now it's tacky, cheesy, and just relentless self-promotion.
Again, like I said before, it's also a terrible benchmark.
I was expecting something more realistic... the true test of what you are doing is how representative is the thing in relation to the real world. E.g. does the pelican look like a pelican as it exists in reality? This cartoon stuff is cute but doesnt pass muster in my view.
If it doesn't relate to the real world, then it most likely will have no real effect on the real economy. Pure and simple.
In contrast, the only "realistic" SVGs I've seen are created using tools like potrace, and look terrible.
I also think the prompt itself, of a pelican on bicycle, is unrealistic and cartoonish; so making a cartoon is a good way to solve the task.
It’s impossible for it to do anything but cut code down, drop features, lose stuff and give you less than the code you put in.
It’s puzzling because it spent months at the head of the pack now I don’t use it at all because why do I want any of those things when I’m doing development.
I’m a paid subscriber but there’s no point any more I’ll spend the money on Claude 4.6 instead.
Me: Remove comments
Literally Gemini: // Comments were removed
Gemini has been way behind from the start.
They use the firehose of money from search to make it as close to free as possible so that they have some adoption numbers.
They use the firehose from search to pay for tons of researchers to hand hold academics so that their non-economic models and non-economic test-time-compute can solve isolated problems.
It's all so tiresome.
Try making models that are actually competitive, Google.
Sell them on the actual market and win on actual work product in millions of people lives.
HN guidelines prefer the original source over social posts linking to it.
"The price" is the marginal price I am paying on top of my existing Google 1, YouTube Premium, and Google Fi subs, so basically nothing on the margin.