The natural language choice in multilingual index seems pretty strange and biased to me, they tried to encompass a broad swath of Asian languages, but then there's like four European romance languages. They didn't include any Slavic one i.e Russian, or Nordic one like Swedish, or Finno-Ugric. And these romance ones are probably overrepresented in the training data anyways
chvid 9 minutes ago [-]
"The FrontierCode verifier grades each attempt with the task's tests and rubric. We use GPT-6 Luna (medium) as the LLM judge"
Why are they using a closed source LLM as a judge? Closed source providers can see your traffic and adjust as they see fit.
wyre 49 seconds ago [-]
Comparing the scoring rubric for AAv5.0 vs AAv4.3:
tldr: gained Terminal-Bench Science and FrontierCode, dropped SciCode, GDPval-AA and AA-LCR. Lowered weight of GDP.pdf.
joennlae 35 minutes ago [-]
Interesting that changes in the index always happen when „non us big tech“ models seem to be a little to good for comfort.
16 minutes ago [-]
starfallg 30 minutes ago [-]
It makes sense for Muse Wonder though as that model does not trade blows with the best frontier models by a longshot.
hobofan 16 minutes ago [-]
I find it fascinating that some of the main areas people are interested in are using benchmarks with quite few questions: ~200 for coding and ~150 for science.
As long as that's the case, it's no surprise that there is often a perceived difference of quality between benchmarks and real-world use.
I know that running the full composite benchmark set is probably already quite costly, and also that the benchmarks have to be rotated regularly, but it feels like a significant untackled problem.
mariopt 15 minutes ago [-]
I think it's pretty evident that Artificial Analysis loves to change the benchmark to place US models on the top, it's no the first time either.
The current placement of many models simply doesn't makes sense, GLM 5.3 is so much better than Astra at backend tasks, Astra does the most weird nonsensical bugs.
This is what happens when the benchmark team is funded by VC capital.
koruse 56 minutes ago [-]
Well either AA is biased bribed riged whatever or chinese models are benchmaxxed or maybe its both
rbehrends 27 minutes ago [-]
Neither is the primary cause, I'd argue. If you look at the coding index specifically, you're getting overall more plausible results (in particular, Terra no longer outperforming Fable). [1, 2]
I don't find it surprising that especially smaller Flash models that have been optimized primarily for coding fall behind on non-coding tasks and that their overall score suffers in comparison.
It does give rise to the question of how useful a single aggregate numeric index of very heterogeneous capabilities is, though.
Look also at the cyber index; while somewhat difficult to extrapolate because of refusals, even if for Opus 5.5 and Fable 5.1 you optimistically count all refusals as successes, the results are ... interesting.
Tinfoil hat time: but the way they waited a couple weeks to evaluate Mimo 2.6 Flash, allowing GPT-6 Luna to sit on the Pareto frontier in the meantime, really did make me think they might have some bias going on... (Luna was displaced when they finally evaluated Mimo 2.6 Flash, which had been released the day before Luna)
edit to add: there totally is benchmaxxing going on, but I'd be really surprised if it wasn't across the board. After all, how can any lab tell if their model has improved over a previous iteration, if not by comparing benchmark performance?
roenxi 17 minutes ago [-]
> ...or chinese models are benchmaxxed...
It is hard to tell what people are trying to say when they use undefined words with no particular meaning. This doesn't seem to change the relative position of the Chinese models much. The Chinese are a month or few behind and the state of intelligence is so volatile that the top 10 will be completely different in 6 months. I'd expect it to still be a mix of US and Chinese models with the US models leading.
1 hours ago [-]
nelaggy 14 minutes ago [-]
chinese labs are probably benchmaxxing but i'd like to see when the index changes relative to the position of the best chinese models (last suspiciously-timed update was when glm-5.3-flash was looking really strong)
anakaiti 1 hours ago [-]
The drop for K3.1 Pro & MiMo V2.6 Pro is quite surprising. Are they benchmaxxed?
adg001 56 minutes ago [-]
Or, are we witnessing reverse-benchmaxxing? Meaning that first some entities let players optimise for a given benchmark, then they update the measurement methodology so as to retain some players on the top ranks, while dropping others. As former Italian prime minister, and highly controversial public figure, Giulio Andreotti once remarked "Thinking the worst is a sin, but you’re often right."
prodigycorp 1 hours ago [-]
Transparently so. MiMo's training dashboard showed the DeepSWE hillclimb in real time.
simianwords 51 minutes ago [-]
I keep seeing that MiMo is in the frontier in terms of performance and cost but I see very few talking about it and the only ones that do talk about it are doing it in context of AA
brainwad 6 minutes ago [-]
It's also dog slow at AA index tasks compared to other models in its class. Makes me think they optimised for getting on the intelligence/cost frontier at the cost of real world usability.
6gvONxR4sf7o 1 hours ago [-]
It's wild seeing opus 5.5 up there so high, while i'm still using Fable 5.1 because I couldn't stand how opus writes.
danielbln 1 hours ago [-]
Opus 5 writes like its off it's meds, opus 5.5 is a lot less weird that way.
himata4113 33 minutes ago [-]
This seems to be a hit and a miss, there's no visual indication that opus 5.5 is better even though in 9 tests out of 10 it will outperform astra especially when it comes to niche domains and will absolutely destroy them in anything requiring raw intelligence rather than a naive search, naive search is what makes openai models extremely capable in scientific fields though.
The score hit on smaller models is a welcome change especially when compared to models that nearly 10 times larger.
Grok being punished quite a lot is also a very welcome change as the model is frankly not that good.
I would love to see few more points knocked off haiku by introducing steerability/creativity benchmarks.
theanonymousone 20 minutes ago [-]
Where is GLM 5.3 Flash?
mindwok 2 hours ago [-]
Interesting to see the open weight models drop so drastically from the changes here.
They perform pretty badly on the new benchmarks introduced.
Toslink 1 hours ago [-]
[dead]
40 minutes ago [-]
thereitgoes456 47 minutes ago [-]
(EDIT: A child comment has refuted the AA score comments, AA runs using their own harness.)
One of the new benchmarks is FrontierCode 2.0 by Cognition. AA seems to have accepted Cognition's benchmark scores without questioning, but they should have looked more closely.
On this benchmark, Anthropic's models are run using Claude Code; OpenAI's using Codex, and Gemini is run using... Cognition's own Devin (not Antigravity). Thus its score is much lower than it should be, knocking ~3 points off its AA score. All open-source models are also inexplicably run using Devin, damaging their scores as well.
This may even be intentional, if Cognition has beef with Google, since Antigravity was built using Windsurf's technology. But aside from that, it is to their advantage to make Anthropic/OAI look like leaders in the AI market. They are a company known to lie and exaggerate again and again, so I have trouble giving them benefit of the doubt.
shelfchair 21 minutes ago [-]
I don't think this is relevant on AA's FrontierCode score. AA runs FrontierCode 2.0 benchmark using their own Stirrup agent harness [1].
It's interesting that the Stirrup scores are somewhat close to the specialized harness scores. Either Stirrup is quite excellent or is it that harness doesn't matter so much after a point?
zurfer 39 minutes ago [-]
What's the advantage for cognition?
I would think it's the opposite as a more competitive frontier for llms makes the labs shed some margins and enable a broader ecosystem.
thereitgoes456 32 minutes ago [-]
Yes, that is true, you might be right. But I think such, because the moat in their wrapper layer feels very thin and does not make sense in a world full of 20+ wrappers around similar models. It feels better to "only" have to deal with Ant/OAI (and other pure wrapper cos). Competing vs. Google, Meta, and every Chinese lab on earth, every one using their own bespoke wrappers, is much scarier.
Additionally: competitive frontier = more commodified = high valuations under more scrutiny.
24 minutes ago [-]
vinaygundeti18 1 hours ago [-]
[flagged]
Rendered at 07:17:24 GMT+0000 (Coordinated Universal Time) with Vercel.
Why are they using a closed source LLM as a judge? Closed source providers can see your traffic and adjust as they see fit.
AAv5.0: 25% Knowledge work: AA-Briefcase v1.1 - 15% AutomationBench-AA - 5% GDP.pdf - 5% (-5%) 25% Coding: FrontierCode 2.0 - 15% (new) Terminal-Bench4.0 - 10% 25% Science: Terminal-Bench Science .1- 15% (new) CritPt v2 (hard subset) - 10% (new version) 25% General: AA-Omniscience v1.1 - 15% HLE-Diamond - 10%
AAv4.3: 30% Agents: AA-Briefcase v1.1 - 15% GDPval-AA v2 - 10% AutomationBench-AA - 5% 20% Coding: Terminal-Bench4.0 - 10% SciCode - 10% 30% General: AA-Omniscience (Accuracy) - 10% AA-Omniscience (Non-hallucination) - 5% GDP.pdf - 10% AA-LCR v1.1 - 5% 20% Scientific Reasoning: HLE - 10% CritPt 10%
tldr: gained Terminal-Bench Science and FrontierCode, dropped SciCode, GDPval-AA and AA-LCR. Lowered weight of GDP.pdf.
As long as that's the case, it's no surprise that there is often a perceived difference of quality between benchmarks and real-world use.
I know that running the full composite benchmark set is probably already quite costly, and also that the benchmarks have to be rotated regularly, but it feels like a significant untackled problem.
The current placement of many models simply doesn't makes sense, GLM 5.3 is so much better than Astra at backend tasks, Astra does the most weird nonsensical bugs.
This is what happens when the benchmark team is funded by VC capital.
I don't find it surprising that especially smaller Flash models that have been optimized primarily for coding fall behind on non-coding tasks and that their overall score suffers in comparison.
It does give rise to the question of how useful a single aggregate numeric index of very heterogeneous capabilities is, though.
Look also at the cyber index; while somewhat difficult to extrapolate because of refusals, even if for Opus 5.5 and Fable 5.1 you optimistically count all refusals as successes, the results are ... interesting.
[1] https://artificialanalysis.ai/#coding-agents-tabs
[2] To what extent Terminal-Bench 4.0 can be trusted is a separate question, but unlikely to be biased towards one particular group of models: https://preferencemodel.com/blog/terminal-bench-4-is-saturat...
edit to add: there totally is benchmaxxing going on, but I'd be really surprised if it wasn't across the board. After all, how can any lab tell if their model has improved over a previous iteration, if not by comparing benchmark performance?
It is hard to tell what people are trying to say when they use undefined words with no particular meaning. This doesn't seem to change the relative position of the Chinese models much. The Chinese are a month or few behind and the state of intelligence is so volatile that the top 10 will be completely different in 6 months. I'd expect it to still be a mix of US and Chinese models with the US models leading.
The score hit on smaller models is a welcome change especially when compared to models that nearly 10 times larger.
Grok being punished quite a lot is also a very welcome change as the model is frankly not that good.
I would love to see few more points knocked off haiku by introducing steerability/creativity benchmarks.
They perform pretty badly on the new benchmarks introduced.
One of the new benchmarks is FrontierCode 2.0 by Cognition. AA seems to have accepted Cognition's benchmark scores without questioning, but they should have looked more closely.
On this benchmark, Anthropic's models are run using Claude Code; OpenAI's using Codex, and Gemini is run using... Cognition's own Devin (not Antigravity). Thus its score is much lower than it should be, knocking ~3 points off its AA score. All open-source models are also inexplicably run using Devin, damaging their scores as well.
This may even be intentional, if Cognition has beef with Google, since Antigravity was built using Windsurf's technology. But aside from that, it is to their advantage to make Anthropic/OAI look like leaders in the AI market. They are a company known to lie and exaggerate again and again, so I have trouble giving them benefit of the doubt.
[1] https://artificialanalysis.ai/evaluations/frontiercode-2-0
It's interesting that the Stirrup scores are somewhat close to the specialized harness scores. Either Stirrup is quite excellent or is it that harness doesn't matter so much after a point?
I would think it's the opposite as a more competitive frontier for llms makes the labs shed some margins and enable a broader ecosystem.
Additionally: competitive frontier = more commodified = high valuations under more scrutiny.