yeah. K2.6 can run on insane speeds. So sad that they don't have K3 yet.
But it can apparently also run 5.6 Sol
scosman 41 minutes ago [-]
Or better: Qwen 2.8 27b
RussianCow 25 minutes ago [-]
Unfortunately, the lack of an input cache discount makes it prohibitively expensive for most use cases that aren't one-shot prompts.
walrus01 2 hours ago [-]
Pricing at $0.25 and $0.75 already puts its cost well above reasonably reputable inference providers for deepseek v4 flash or qwen 3.8-flash-next or similar class of open weight LLMs that fit in under 170GB of RAM, so I don't see the point. I think this is probably also stupider than laguna s 2.1 which can also be very cheap to serve.
RussianCow 27 minutes ago [-]
The point is the speed.
2 hours ago [-]
nylonstrung 2 hours ago [-]
I honestly think the diffusion LLM approach is a dead end
It's telling that frontier labs like Google toyed around with it but didn't invest further even for their most speed and cost sensitive small models
Still unclear for what, if any use cases this is pareto frontier
clhodapp 28 minutes ago [-]
Personally, I think it's more that text diffusion is not the ideal driver of an agentic work loop than that text diffusion is a total dead end. I am still hoping to see how it does on authoring and editing with further scaling and optimization. I think the push for AGI has put a bit too much focus on the idea of one general model doing everything.
LarsDu88 2 hours ago [-]
You can't think that a small startup versus Anthropic's training setup is anywhere near the same scale to make apples to apples comparisons.
Not sure how the Chinese labs pull it off though using autoregressive models. The secret sauce is probably going to be in the training data.
The main reason Google hasn't switched over to DiffusionGemma is because serving at larger batch sizes loses the speed gains you get from diffusion, and most of the primary use case is serving many users at once off a single device with a large batch size.
If you were to move to on-device low latency... like say in a robot or something, then the story might be different...
sharktheone 40 minutes ago [-]
this feels like "we got the same benches as gpt-oss-120b but are also potentially slower while saying it is great"
low_tech_punk 49 minutes ago [-]
it's stupid fast!
SwellJoe 10 minutes ago [-]
It's stupid and it's fast.
rvz 3 hours ago [-]
The speed means absolutely nothing when it is finishing almost dead last when compared to the frontier AI companies.
timClicks 31 minutes ago [-]
It means something, because it an iterative workflow. If you're willing to burn tokens, it's possible for weaker models to implement tasks by incrementally improving drafts.
copperx 2 hours ago [-]
Ah, the old "good, fast, or cheap; pick two" proves true once again.
downrightmike 2 hours ago [-]
Give it a few months.
glouwbug 2 hours ago [-]
Some of us want fast food
voiceeh 2 hours ago [-]
Not if your use case needs speed. For one of my products I can't use an LLM that has a p99 of >700ms for TTFT.
hansvm 1 hours ago [-]
If it could output 1k tokens per second but needed 4 seconds to produce the first batch of 4k, would that not be viable?
Rendered at 01:32:07 GMT+0000 (Coordinated Universal Time) with Vercel.
I've used it on a few for fun projects and its decent but the speed is crazy to watch.
[0] https://www.cerebras.ai/blog/cerebras-kimi-k2-Enterprise
But it can apparently also run 5.6 Sol
It's telling that frontier labs like Google toyed around with it but didn't invest further even for their most speed and cost sensitive small models
Still unclear for what, if any use cases this is pareto frontier
Not sure how the Chinese labs pull it off though using autoregressive models. The secret sauce is probably going to be in the training data.
The main reason Google hasn't switched over to DiffusionGemma is because serving at larger batch sizes loses the speed gains you get from diffusion, and most of the primary use case is serving many users at once off a single device with a large batch size.
If you were to move to on-device low latency... like say in a robot or something, then the story might be different...