It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website
simonw 11 minutes ago [-]
The pelicans are still all rubbish. If they make it into the training set it doesn't help the models produce better pelicans, if anything it will make them perform worse!
cebert 2 hours ago [-]
Simon has stated a few times that he knows it’s possible that pelicans could be in the training sets. He also has other tests he doesn’t share publicly. He’s just a fan of pelicans.
hungryhobbit 34 minutes ago [-]
From the article it doesn't even sound like he cares about pelicans at all, and doesn't think they are a good way to compare models anymore ... but people are used to seeing the test now, and it does serve as a common "hello world" unit of work.
eminence32 2 hours ago [-]
Pelicans and bikes can be in the training set without them training for this specific benchmark.
j_maffe 1 hours ago [-]
Yes and that would improve its ability to draw SVGs of pelicans on bikes, no?
14 minutes ago [-]
barrenko 25 minutes ago [-]
Would it? Tongue in cheek.
asasidh 58 minutes ago [-]
and that is bad because ?
program_whiz 50 minutes ago [-]
the nature of the test was to see if the models can effectively compose an image of a novel concept outside the training set. If they are trained on it, it ceases to be an interesting test to some extent.
wasabi991011 39 minutes ago [-]
I would urge you to re-read the blog post you are commenting on. It pretty clearly explains how it is an interesting test independently of "see[ing] if the models can effectively compose an image of a novel concept outside the training set".
cyanydeez 45 minutes ago [-]
it's still interesting because there's no pelican-on-bike model, and if you're training a model well enough, then it should be obvious when a model has reached "AGI" or whatever.
podgietaru 2 hours ago [-]
More to it, the actual bloody companies are using them as a reference. Maybe it’s a 3d version, not an svg - but it clearly shows they’re on the radar of these companies.
HardCodedBias 27 minutes ago [-]
A person from Google famously put on her linkedin that her job was to optimize SVG for Gemini 3.0.
semilin 2 hours ago [-]
They can be in the training set but not deliberately trained for. There may be a lot of people posting pelican svgs, but not typically because they're high quality and worth replicating.
andy_xor_andrew 1 hours ago [-]
Did you read the post? It's not even that long. He explicitly mentions this...
Barbing 1 hours ago [-]
Are they responding to: “I’m still not convinced that labs are training for the benchmark—if they were, I’d expect much better results.”
Topfi 31 minutes ago [-]
Respectfully, did you? The comment was specific to doubting the believe simonw has that labs are not training [0] specifically for this task, which is exactly what simonw wrote in the post [1], that it is a believe of his that they don't. He did not mention any kind of evidence or any piece of information that would indicate that the commenter didn't read the blog post.
Did you read either the post or the comment it was referencing?
On the note of training on SVGs, I have seen some labs models outperform when prompted for SVGs of certain animal and action combinations (pelican on bike, panda eating burger, etc.) compared to other similarly outlandish prompts for SVG output that are not part of widely reported benchmarks, even shared evidence one of the last times this came up on here.
[0] ... incredible Simon still believes ...
[1] I’m still not convinced that labs ....
drcongo 49 minutes ago [-]
Clearly not. There's a subset of HN users who rush to post this same thing every single time.
Topfi 26 minutes ago [-]
Maybe it gets posted every time because besides a personal believe by the person popularising this "benchmark", there is no reason to assume that certain labs aren't intentionally training to game this and every other lab at least unintentionally gets improvements for this specific combination of animal and action because the internet is full of both good and bad examples, often ranked, which does inevitably become training data.
I have shared examples of certain models by certain labs doing far better on the pelican cycling vs other, similar prompts. Just operating on a feeling that labs don't optimise for this (as mentioned, even if they don't training data is filled with these) is not solid enough that criticism shouldn't be leveraged when it comes up.
simonw 9 minutes ago [-]
> I have shared examples of certain models by certain labs doing far better on the pelican cycling vs other, similar prompts
Please share those again!
One of the things I'm most looking forward to is a lab producing a model that creates a really great pelican riding a bicycle and then a terrible sloth riding a skateboard (or whatever).
I've not seen that myself yet.
oceanplexian 1 hours ago [-]
Imagine if we applied this train of logic to humans.
"That artist saw a pelican at the beach once!" [cue the outrage] "He's not a real artist, he's a cheater and produces nothing original!"
program_whiz 47 minutes ago [-]
This is a sight-reading test. If a musician practices a piece for thousands of hours, it would no longer be an effective sight reading / creativity test. The purpose of the test was to see how models would compose something novel requiring the ability to compose orthogonal, normally unrelated, components into a coherent image.
alexjplant 27 minutes ago [-]
We do. People who, for example, memorize question banks to pass certification tests without knowing the underlying material are equally frowned upon for not having the problem solving skills that they purport to.
I'll leave the contrasts between LLMs and people to the well-written sibling comments.
computably 1 hours ago [-]
Except, of course, LLMs are not humans, and they do not learn or "reason" in a way which even remotely resembles humans.
Plus obviously humans can still overfit to a specific style of test.
devttyeu 2 hours ago [-]
> How does the prompt “Generate an SVG of a pelican riding a bicycle” add up to 95 input tokens? OpenAI’s tokenizer counts 10, Anthropic’s counts 10 for Opus 4.6, 30 for Opus 4.7 and 25 for Sonnet 5/Fable 5. Prompting “hi” to Kimi K3 counted 86 tokens, suggesting there may be an 85 token hidden system prompt. It refused to leak it though.
Like Simon concludes the article, the main use of this isn't to say which model is "better", but to try and poke at the model to sort out things like quality vs cost vs speed.
So I put together a quick comparison of the last couple iterations of Opus, Fable and now Kimi.
It will be valuable to have two types of benchmarks: ones that evolve alongside the models and ones that never change. You probably can't get historical stability and resistance to flooding and training on at least some parts of it from the same test
andai 15 minutes ago [-]
3T is impressive, but parameter count seems to be less important than I thought.
GLM is half the size of DeepSeek but costs four times as much, and beats it on every benchmark.
I'm not an expert on this stuff but it seems to be the attention mechanism. DeepSeek were bragging about how cheap they made it. But if you cut costs on attention you get worse results with way more parameters.
If I had to guess it seems to be the difference between memory (params) and intelligence (attention density). I think you need both.
Zsfe510asG 2 minutes ago [-]
Kimi is right out since they use classical music branding to sell their slop. At least McDonalds does not sell Verdi or Allegro burgers.
Why does Kimi not use a "Double Cheese Whammy" branding for "their" butchered and stolen IP?
bcit-cst 42 minutes ago [-]
The gap is closing . I think Kimi 3 is only 3 months behind the US model. It’s gpt 5.5 class model , which was released in the end of April.
rdtsc 58 minutes ago [-]
The idea is not to use pelicans on bikes but a similarly random non-sensical prompts: crows on scooters, squirrels in a moon rover etc. Then pick another one for another for next cross-llm evaluation.
tibbar 57 minutes ago [-]
I wonder how the Chinese labs are training a 3 trillion parameter model on what has to be vastly smaller compute resources. If the U.S. compute advantage is persistent, it's hard to imagine that Chinese labs will be able to keep pace forever, as a matter of physics, but... so far they seem to be doing just fine.
dopa42365 9 minutes ago [-]
It's not like same parameter count models are identical, so that doesn't appear to be an indicator for quality, or even compute requirements?
There seems to be more to producing a better model than brute forcing parameter count after all.
Lerc 2 hours ago [-]
Do any of the vision models render the SVG and look at the result.
Perhaps more importantly can they do that during reinforcement training. Learning how to critically analyse the appearance of what it generates would be quite useful.
Manually feeding images back to models has been hilariously bad in the past which suggests that relating something it sees to something it wrote is not an ability it is very good at.
cherioo 27 minutes ago [-]
I imagine all vision models have to do this, this being html rendering, to be able to do well in web design.
mesmertech 2 hours ago [-]
My personal benchmark for new models has been to compare video making skills with something like remotion. Usually reveals if they have any "taste" or outside the box thinking.
I'm starting to not trust any "benchmarks" when it comes to frontier models at least. As an example Sol feels the most "gets stuff done" but has zero taste, or any capability to surprise.
And for frontier models I go one step ahead and try to recreate a complex animation video, with the ability for the model to review its own work. And at this Fable is still the top one.
Ex: https://www.youtube.com/watch?v=uDAeAuYyl0E (recreation of Claude announcement video) and https://www.youtube.com/watch?v=cSsVNtGPOIg (recreation of a fireship video). Sol did something similar but you can instantly tell its AI slop from very small things, and it just has no narrative or thought put into the writing.
And on creativity at least visually, Gemini 3.1 pro is somehow still up there. But its really hindered by its inability to use tool calls effectively or make a long term plan.
nothercastle 43 minutes ago [-]
It’s not bad kind of expensive for 25c but if the prompt is rendered cost is much better.
criddell 24 minutes ago [-]
I wonder what the non-subsidized cost is. Add in the electricity and water too.
We may be boiling the oceans but at least we are finally getting some good SVGs of pelicans on bicycles.
whywhywhywhy 1 hours ago [-]
Don't see why we have to have this spammed every model release when Fable class models perform the same as Opus on basic tasks like these.
dgellow 25 seconds ago [-]
What spam? It’s one article. You can skip it
hkalbasi 2 hours ago [-]
Is there a gallery of all pelicans generated by simon over time?
If Simon reads this debate, I would gladly vote for such a gallery. It would belong to "digital heritage of mankind".
Xx_crazy420_xX 2 hours ago [-]
I would be surprised if pelican svgs are not part of the training corpus rn
skeledrew 2 hours ago [-]
If that were the case then it'd do a way better job. Think experienced artist level.
teravor 40 minutes ago [-]
how would great pelicans make their way into the training set?
what they do have are many different pelicans and people helpfully rating them in the comments.
seventeengivens 2 hours ago [-]
[dead]
dsign 3 hours ago [-]
Another day, another model and another pelican :-)
I can't help but wonder where is the trend going? What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing? Or maybe the prompt then will be "make a pelican ride a bicycle", and out will come the genetic code for a giant pelican with extremities suitable for a handle bar and pedals, and an inborn affinity to ride bicycles?
rvz 2 hours ago [-]
You are thinking too hard on this. This entire "benchmark" is a performative joke for attention that only works on HN.
> What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing?
We will just have more of the same.
simonw 7 minutes ago [-]
> This entire "benchmark" is a performative joke for attention that only works on HN.
I take exception to that! It's a performative joke for attention that works far more widely than just Hacker News.
Yiin 2 hours ago [-]
You say it's performative joke, but it all depends what you're using model for. So far the rule has been quite straightforward, better models consistently renders pelican in higher quality, I've yet to see an exception. It is also a good enough (for me at least) test for "taste" the model has.
j_maffe 1 hours ago [-]
> better models consistently renders pelican in higher quality
The article literally avoid making this argument and gives counterexamples to this statement.
ofjcihen 2 hours ago [-]
I’m excited for this specific brand of survival horror.
kherud 2 hours ago [-]
Imagine what amazing SVG generators we could have if Simon had randomized the target image from the start (and companies wouldn't just overfit on pelicans).
You still need an OpenRouter API Key and be careful this can burn quite a bit of money.
hdjdjdjdjdjdjd 3 hours ago [-]
[flagged]
brcmthrowaway 1 hours ago [-]
Imagine shilling some CLI tools no one uses in this post.
dghlsakjg 1 hours ago [-]
Lighten up.
You’re reading a personal blog and complaining about an open source personal project he runs and distributes for free. He’s allowed to talk about his personal work on his personal blog. Especially considering the cli utility he talks about is directly related to the post.
Imagine complaining about someone generating valuable content for free and not packaging it to your personal tastes.
mrcwinn 2 hours ago [-]
K3 is as expensive as Sonnet, not great at writing English, is handing IP back to the Chinese, and once open source will be difficult to run at scale without the compute that OpenAI and Anthropic have largely grabbed.
Sorry, how again is this the end of the frontier labs?
rootlocus 2 hours ago [-]
According to some benchmarks has the coding capability of Opus at the price of Sonnet, supposedly will be open weights and is not subject to random trade wars with allied states.
Competition is always good.
olig15 2 hours ago [-]
You mean the scale that AWS provides with Bedrock?
nickthegreek 55 minutes ago [-]
Bedrock needs to actually update their chinese models to the newest versions for this to matter.
BugsJustFindMe 2 hours ago [-]
> This is expensive—the pelican cost 25 cents!
Engineers get unbelievably silly about evaluating costs of things.
"The tokens are so expensive!" Oh my sweet child, how much would even the least capable human effort cost? This is what the executives properly understand that the programmers don't.
Yiin 2 hours ago [-]
they're comparing to similar capability llm models, not humans. If one dishwasher does job at similar quality as another dishwasher, but using 30% more water and energy, you wouldn't compare to how much it costs human to do the same work, it would make no sense.
BugsJustFindMe 2 hours ago [-]
> they're comparing to similar capability llm models, not humans
25 cents is 10x the cost of 2.5 cents, but it's still extremely cheap for the product. It's very much the wrong comparison for a world where the primary competition is still humans who need to eat, and it treats percentage differences as more important than absolute differences when the opposite is true.
jchw 2 hours ago [-]
Well first of all, any non-trivial use of LLMs is going to be orders of magnitude more tokens than this, usually multiple millions at minimum. Benchmarks are just benchmarks after all.
Secondly, humans vs LLMs are apples vs oranges. It makes no more sense to compare human costs vs LLM costs as it would have to compare human costs vs calculator costs. LLMs are faster and cheaper but extremely different beasts with different limitations. Humans do not one-shot SVGs of pelicans riding bicycles, and they do not charge in tokens.
Comparing LLM cost efficiency is not something that should need to be defended. It's quite straightforward and reasonable...
bakugo 2 hours ago [-]
Would anyone pay a human to create an SVG of a pelican riding a bike?
BugsJustFindMe 2 hours ago [-]
In fact humans get paid to create SVGs of all kinds of things.
codezero 1 hours ago [-]
Well, no, not now they won’t.
Rendered at 18:43:19 GMT+0000 (Coordinated Universal Time) with Vercel.
Did you read either the post or the comment it was referencing?
On the note of training on SVGs, I have seen some labs models outperform when prompted for SVGs of certain animal and action combinations (pelican on bike, panda eating burger, etc.) compared to other similarly outlandish prompts for SVG output that are not part of widely reported benchmarks, even shared evidence one of the last times this came up on here.
[0] ... incredible Simon still believes ...
[1] I’m still not convinced that labs ....
I have shared examples of certain models by certain labs doing far better on the pelican cycling vs other, similar prompts. Just operating on a feeling that labs don't optimise for this (as mentioned, even if they don't training data is filled with these) is not solid enough that criticism shouldn't be leveraged when it comes up.
Please share those again!
One of the things I'm most looking forward to is a lab producing a model that creates a really great pelican riding a bicycle and then a terrible sloth riding a skateboard (or whatever).
I've not seen that myself yet.
"That artist saw a pelican at the beach once!" [cue the outrage] "He's not a real artist, he's a cheater and produces nothing original!"
I'll leave the contrasts between LLMs and people to the well-written sibling comments.
Plus obviously humans can still overfit to a specific style of test.
This is quite possibly reasoning-effort prompt which is injected before the opening <think> token whenever you set a custom reasoning effort, see e.g. DeepSeek-V4 max mode prompt: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main...
So I put together a quick comparison of the last couple iterations of Opus, Fable and now Kimi.
Kimi is cheapest by 5x but also slowest by 2x
https://9gpyw4uxr2.evvl.io/
GLM is half the size of DeepSeek but costs four times as much, and beats it on every benchmark.
I'm not an expert on this stuff but it seems to be the attention mechanism. DeepSeek were bragging about how cheap they made it. But if you cut costs on attention you get worse results with way more parameters.
If I had to guess it seems to be the difference between memory (params) and intelligence (attention density). I think you need both.
Why does Kimi not use a "Double Cheese Whammy" branding for "their" butchered and stolen IP?
There seems to be more to producing a better model than brute forcing parameter count after all.
Perhaps more importantly can they do that during reinforcement training. Learning how to critically analyse the appearance of what it generates would be quite useful.
Manually feeding images back to models has been hilariously bad in the past which suggests that relating something it sees to something it wrote is not an ability it is very good at.
I'm starting to not trust any "benchmarks" when it comes to frontier models at least. As an example Sol feels the most "gets stuff done" but has zero taste, or any capability to surprise.
And for frontier models I go one step ahead and try to recreate a complex animation video, with the ability for the model to review its own work. And at this Fable is still the top one. Ex: https://www.youtube.com/watch?v=uDAeAuYyl0E (recreation of Claude announcement video) and https://www.youtube.com/watch?v=cSsVNtGPOIg (recreation of a fireship video). Sol did something similar but you can instantly tell its AI slop from very small things, and it just has no narrative or thought put into the writing.
https://mesmer.tools/benchmarks/ai-video-generation , I usually put basic ones here.
We may be boiling the oceans but at least we are finally getting some good SVGs of pelicans on bicycles.
what they do have are many different pelicans and people helpfully rating them in the comments.
I can't help but wonder where is the trend going? What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing? Or maybe the prompt then will be "make a pelican ride a bicycle", and out will come the genetic code for a giant pelican with extremities suitable for a handle bar and pedals, and an inborn affinity to ride bicycles?
> What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing?
We will just have more of the same.
I take exception to that! It's a performative joke for attention that works far more widely than just Hacker News.
You still need an OpenRouter API Key and be careful this can burn quite a bit of money.
You’re reading a personal blog and complaining about an open source personal project he runs and distributes for free. He’s allowed to talk about his personal work on his personal blog. Especially considering the cli utility he talks about is directly related to the post.
Imagine complaining about someone generating valuable content for free and not packaging it to your personal tastes.
Sorry, how again is this the end of the frontier labs?
Competition is always good.
Engineers get unbelievably silly about evaluating costs of things.
"The tokens are so expensive!" Oh my sweet child, how much would even the least capable human effort cost? This is what the executives properly understand that the programmers don't.
25 cents is 10x the cost of 2.5 cents, but it's still extremely cheap for the product. It's very much the wrong comparison for a world where the primary competition is still humans who need to eat, and it treats percentage differences as more important than absolute differences when the opposite is true.
Secondly, humans vs LLMs are apples vs oranges. It makes no more sense to compare human costs vs LLM costs as it would have to compare human costs vs calculator costs. LLMs are faster and cheaper but extremely different beasts with different limitations. Humans do not one-shot SVGs of pelicans riding bicycles, and they do not charge in tokens.
Comparing LLM cost efficiency is not something that should need to be defended. It's quite straightforward and reasonable...