This is a promising direction! Unfortunately, I think the benchmark result here is essentially meaningless.
I recently discovered this same lesson the hard way. I was trying to get a multi-agent system I was building to improve upon GPQA Diamond scores (system here: http://pellmell.ai). No matter how hard I tried, I could not get any lift. When Fable 5 dropped, it also did not improve upon Opus, and I realized my mistake. The benchmark was saturated!
Now, looking at the result here, I see a similar pattern. Fable is not better than Opus, and the score is ~95%. Notably, this post omits which subagent is being used. Why? An intellectually honest way to tell if this thing really works would be to run that agent and report its score and cost as well.
Going back to my GPQA Diamond lesson, you can see here how a saturated leaderboard behaves https://artificialanalysis.ai/evaluations/gpqa-diamond. Fable gets 92.6% for $0.22 per task while several models score higher for $0.01. I could easily publish a router that “enhances Fable on GPQA Diamond” showing improved score for lower cost, just by implementing a router that picks the model at random!
docheinestages 23 minutes ago [-]
Your landing page is very hard to read. The font size is literally 10px for some content, while animations distract the reader.
Dear YC, please force the startups to dedicate some of the 500k funding for standard web design.
hmokiguess 1 hours ago [-]
> P.S: we hid a code on the website, see if you can unlock the secret page at the footer, all built with Bullet
Is it even hidden when your AI ends up tagging it with `aria-label="Hidden secret code"`? Lol. Fun mini game.
adi1 1 hours ago [-]
haha fair enough, surprisingly people still struggle to find it, glad you found it and liked the game!
myshapeprotocol 2 hours ago [-]
Focusing on execution speed for coding agents is the right bottleneck to tackle. Exciting launch.
# Share chats with Bullet — helps us improve model routing and answer quality
Enabled by default.
alsima 7 hours ago [-]
Thanks for bringing this up and trying it out! We've disabled Cmd+Option+I in the new version and also have a message during signup about sharing chats with Bullet, let us know if you run into any other issues!
poly2it 33 minutes ago [-]
> We've disabled Cmd+Option+I in the new version
Bulletproof.
xrisk 24 minutes ago [-]
Truly built on vibes.
2 hours ago [-]
tontinton 55 minutes ago [-]
Are you freaking kidding me with YC throwing money at something like this? I guess I can fund raise just by having built https://maki.sh, and months ahead of other founders too...
Supermancho 25 minutes ago [-]
I'm not sure why anyone would fund "creating an agent".
OpenCode is competing with Codex and Claude already.
"The bullets" are not any different than other new agent projects. New agent projects pop up and die constantly and it's not from a lack of funding.
docheinestages 23 minutes ago [-]
YCombinator's investments are unfortunately very questionable nowadays.
Rendered at 17:25:51 GMT+0000 (Coordinated Universal Time) with Vercel.
I recently discovered this same lesson the hard way. I was trying to get a multi-agent system I was building to improve upon GPQA Diamond scores (system here: http://pellmell.ai). No matter how hard I tried, I could not get any lift. When Fable 5 dropped, it also did not improve upon Opus, and I realized my mistake. The benchmark was saturated!
Now, looking at the result here, I see a similar pattern. Fable is not better than Opus, and the score is ~95%. Notably, this post omits which subagent is being used. Why? An intellectually honest way to tell if this thing really works would be to run that agent and report its score and cost as well.
Going back to my GPQA Diamond lesson, you can see here how a saturated leaderboard behaves https://artificialanalysis.ai/evaluations/gpqa-diamond. Fable gets 92.6% for $0.22 per task while several models score higher for $0.01. I could easily publish a router that “enhances Fable on GPQA Diamond” showing improved score for lower cost, just by implementing a router that picks the model at random!
Dear YC, please force the startups to dedicate some of the 500k funding for standard web design.
Is it even hidden when your AI ends up tagging it with `aria-label="Hidden secret code"`? Lol. Fun mini game.
Cmd+Option+I > Console > 'allow pasting'
const onboarding = document.querySelector('#onboarding'); const app = document.querySelector('#app'); onboarding.style.setProperty('display', 'none', 'important'); app.inert = false; app.removeAttribute('aria-hidden'); document.querySelector('#prompt')?.focus();
Also, warning:
# Share chats with Bullet — helps us improve model routing and answer quality
Enabled by default.
Bulletproof.
"The bullets" are not any different than other new agent projects. New agent projects pop up and die constantly and it's not from a lack of funding.