I can't tell based on this article if the authors intend to submit this to Arc-AGI for the private / held-out set of games for a verified score. The final section sorta seems like they won't bother because Arc-AGI-3 is "now saturated"
ClassAndBurn 40 minutes ago [-]
Any custom harness for a problem shows that harness engineering is going away. Eventually models will introspect problems, then build custom harnesses tailored to that. Then use and modify the ephemeral harness as required.
Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.
deepfriedbits 5 minutes ago [-]
Indeed. And you can make this case about any tooling at all that is model adjacent.
vessenes 31 minutes ago [-]
Except in this case, it isn't yet smart enough. But I agree, building this capability in is coming, and will be really awesome.
ClassAndBurn 20 minutes ago [-]
It's likely smart enough. It just needs to be told to do it and provided the ability to introspect it. How close could foundational models get to building this harness if explicitly prompted to?
We've only just started training models to use tools. Next, we'll train them to build them. Harness engineering is an ephemeral art.
couscouspie 6 minutes ago [-]
Letting the provider decide for the harness is a terrible idea in my eyes. Outsourcing harnessing is giving up control over the AI and equivalent to abandoning your sovereignty. It is a regression to a pre-enlightenment era.
vessenes 39 minutes ago [-]
To be clear, we’ll want to see how this performs against the hold-out set. If it holds up, though, it’s a big deal, and kind of in line with the vibes this year, which I’d typify as ‘harness matters’. Maybe we’d upgrade to ‘harness matters immensely’ if this can 100% ARC-AGI-3 on existing models (more in the 13% range without this harness).
I’m pretty excited to see what sort of generalization we come to over the next 12 months on the harness side: if it turns out this can be RLed in as ‘consider if building a world model might help here’ and we get this as another native capacity, that will be interesting. If we get 100 of those problem-solving strategies all included, feels like we will see another hurdle cleared in terms of usefulness.
19 minutes ago [-]
stared 2 hours ago [-]
In the spirit of ARC-AGI-3-like challenges, we just tested if frontier AI models are able to solve a lovely puzzle game, Baba Is You: https://quesma.com/blog/baba-is-bench/
A year ago, Sonnet 4 barely solved the first level. Now, both Fable 5 and GPT-5.6 Sol beat the first two stages. GPT 5.2 is slow, but efficient, while Gemini 3.1 Pro and 3.5 Flash struggle.
sva_ 49 minutes ago [-]
I'm wondering what's up with the release of Gemini 3.5 Pro, they keep postponing it. For a while, Google was doing pretty well with their releases.
teravor 46 minutes ago [-]
it looks like what they are doing is using a frontier model to write a simulator for a game and then solve using it.
it's not as impressive as it looks. the goals of Arc-AGI-like constructs is to get an IQ-like figure using raw'ish 2D measurement 'games' in the hope that it would signify something meaningful.
what this harness does is get the model to write a simulator first, it's measuring something entirely different.
vessenes 28 minutes ago [-]
This is classic goalpost movement. Arc-AGI-3 was launched this year with roughly 0.5% success for frontier models. being able to 99% it in less than six months sets a new record for Arc-AGI saturation timeline. Speaking of singularity measures. It is definitely a big deal, not least in that Chollet needs to cancel his summer vacation and write Arc-AGI-4 now.
teravor 24 minutes ago [-]
Arc AGI are simple games, the hardness comes from the input being basically adversarial to LLM training. if you use an LLM scaffold that removes the adversarial part you are measuring something else.
the harness basically outsources the alien nature of what the LLM is asked to do to algorithms it writes. this would actually be impressive if you got it to do that for a much more complicated game than Arc.
with this harness the ARC AGI test becomes a test of whether or not the model can work out the transition rules in a very simple game.
UltraSane 32 minutes ago [-]
The simulator the model builds is comparable to the mental model of the game humans create. It is also much more efficient, GPT 5.6 Sol cost $25,000 to run on ARC-AGI-3
teravor 30 minutes ago [-]
> simulator the model builds is comparable to the mental model of the game humans create
then they should try to use that for a more complicated game than Arc AGI. Arc games are simple by design, if you have the model simulate them they become trivial.
ubermon 18 minutes ago [-]
> Both scores come from a fixed fallback rule: Opus 4.8 and Sol xhigh run first; games scoring below 80 are rerun with Fable 5 and Sol max, respectively, and the higher per-game score is retained.
hmm, this is like pass@n until you get the high watermark? How would this mean anything?
gandalfgeek 55 minutes ago [-]
Big jump for sure, but definitely comes with a giant grain of salt lacking open-sourcing the harness itself and measuring performance on the held-out set.
levocardia 2 hours ago [-]
(1) What does it score on the private test set?
(2) Does this approach generalize to, e.g., Atari or NES games, or is it just hard-coding priors about the games into the model (as Chollet specifically warned was a chronic problem in benchmarks in the original Arc-AGI paper)
nkmnz 11 minutes ago [-]
Where's the code?
scotty79 7 minutes ago [-]
I pretty much predicted this. If such smart models capable of doing math research fail so hard on such simple games the interface is the problem, not the model. Right harness provides a good interface between the problem and the intelligence.
daytonellwanger 1 hours ago [-]
Can someone tell me what the catch is? To outperform the state-of-the-art so drastically would be massive news, and surely the ARC Foundation would have tested this against the private data set, right?
Alifatisk 2 hours ago [-]
What does it mean to reach 99% score on Arc-AGI-3? That the agent is able to tackle difficult problems?
modeless 2 hours ago [-]
It doesn't necessarily mean anything to reach 99% on the public set. All of the public set is known in advance, so it's possible to hardcode rules that make this easy for the models. ARC-AGI-3 is supposed to measure generalization to unseen games, so the only score that matters is the score on the held out private test set that nobody outside the ARC prize foundation has access to. Also, I believe the private set is significantly harder than the public set.
causal 1 hours ago [-]
We need to see private set results, but if this holds then it might represent a breakthrough in other domains as well.
2 hours ago [-]
westurner 2 hours ago [-]
> Schema, the harness we introduce today, reaches 99% on the ARC‑AGI‑3 Public set using Claude Opus 4.8 and Fable 5, and 95.35% using GPT‑5.6 Sol.
Impressive results. Will this translate to coding agents (and training general purpose and for coding LLMs) too?
---
> When Michelson and Morley could not detect the medium light was supposed to wave in, Lorentz took the first route: keep the aether, patch the rules with contraction hypotheses that absorbed the null result. Einstein took the second: in special relativity, he discarded the aether as part of the state and made simultaneity frame-relative, yielding a simple electrodynamics of moving bodies.
BECs, SVT, Superfluid Quantum Gravity
Massful photons are modeled with Proca fields. Like Einstein, Proca was also a student of Minkowski. The Mass-Equivalence principle ~~does not~~ still holds if photons have mass.
> could not detect the medium light was supposed to wave in,
Superfluid Quantum Gravity (Fedi,) says that there is a medium that light waves through; there is not nothing in space, space is a quantum dilatant superfluid with near-zero viscosity.
Rendered at 20:15:06 GMT+0000 (Coordinated Universal Time) with Vercel.
Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.
We've only just started training models to use tools. Next, we'll train them to build them. Harness engineering is an ephemeral art.
I’m pretty excited to see what sort of generalization we come to over the next 12 months on the harness side: if it turns out this can be RLed in as ‘consider if building a world model might help here’ and we get this as another native capacity, that will be interesting. If we get 100 of those problem-solving strategies all included, feels like we will see another hurdle cleared in terms of usefulness.
A year ago, Sonnet 4 barely solved the first level. Now, both Fable 5 and GPT-5.6 Sol beat the first two stages. GPT 5.2 is slow, but efficient, while Gemini 3.1 Pro and 3.5 Flash struggle.
it's not as impressive as it looks. the goals of Arc-AGI-like constructs is to get an IQ-like figure using raw'ish 2D measurement 'games' in the hope that it would signify something meaningful.
what this harness does is get the model to write a simulator first, it's measuring something entirely different.
the harness basically outsources the alien nature of what the LLM is asked to do to algorithms it writes. this would actually be impressive if you got it to do that for a much more complicated game than Arc.
with this harness the ARC AGI test becomes a test of whether or not the model can work out the transition rules in a very simple game.
hmm, this is like pass@n until you get the high watermark? How would this mean anything?
Impressive results. Will this translate to coding agents (and training general purpose and for coding LLMs) too?
---
> When Michelson and Morley could not detect the medium light was supposed to wave in, Lorentz took the first route: keep the aether, patch the rules with contraction hypotheses that absorbed the null result. Einstein took the second: in special relativity, he discarded the aether as part of the state and made simultaneity frame-relative, yielding a simple electrodynamics of moving bodies.
BECs, SVT, Superfluid Quantum Gravity
Massful photons are modeled with Proca fields. Like Einstein, Proca was also a student of Minkowski. The Mass-Equivalence principle ~~does not~~ still holds if photons have mass.
(edit) Energy-momentum relation: https://en.wikipedia.org/wiki/Energy%E2%80%93momentum_relati...
> could not detect the medium light was supposed to wave in,
Superfluid Quantum Gravity (Fedi,) says that there is a medium that light waves through; there is not nothing in space, space is a quantum dilatant superfluid with near-zero viscosity.