Sabastian Raschka is one of the great LLM researchers/authors. I highly recommend his substack
alealvarezarg 27 minutes ago [-]
Great breakdown. After using Kimi extensively, it's fascinating to see how architectural choices like KDA and NoPE translate into such strong real-world performance. Really impressive engineering.
gokohl 3 hours ago [-]
[flagged]
thunderbird120 2 hours ago [-]
Kimi Delta Attention (KDA), despite having "Attention" in the name, isn't really attention at all in any conventional sense. It's more like an RNN which can be efficiently parallelized during training. It's a very small modification to Gated DeltaNet, which can be described as an RNN whose hidden state acts like a small, editable attention memory.
Because it's RNN-like, it has an inherent idea that X comes before Y which comes before Z in the sequence XYZ. Transformers, by default, don't have that. They operate on sets, unordered collections of unique items. They have no idea where those items are in relation to each-other so you have to clue them in.
Because There are 3 KDA layers per attention layer, and 3 KDA layers before the first attention layer, every single token position is going to be able to learn information about where it is in the sequence before the first actual attention layer.
RoPE is actually a bit destructive, so being able to omit it like this is very convenient. Models like Gemma-4 have a similar structure with 5:1 Sliding Window Attention (SWA) layers for every global attention layer. These are cheap, shitty attention layers which handle local information and which go in-between the big powerful ones, KDA serves the same role in this model. In Gemma only the SWA layers have RoPE while the global attention layers omit it. SWA is actual attention, even if it only operates on a small sliding window, so it needs the positional embedding. KDA isn't, so it doesn't.
>Curious to see if it holds up at frontier scale.
I don't know how much more frontier scale you can get than this, but yes, there's no reason why that wouldn't work at larger scales. Honestly, more parameters just makes it easier for the KDA layers to communicate that positional information better.
wongarsu 3 hours ago [-]
Is frontier scale larger than this? Kimi K3 seems to benchmark in the same range as Opus and Fable. I would have expected they are all in the 2-4T range, with quality of the training and architecture differences as the major differentiators
porridgeraisin 3 hours ago [-]
The number of active parameters is vastly different. Deepseek CEO hinted that he estimates it as an order of magnitude difference in one of his recent interviews.
> Seems to benchmark
yes, but in human usage the differences show up
kamranjon 3 hours ago [-]
Would you happen to have a link to that interview? Sounds like an interesting read.
It has some weird side effects though. for example KV-caches are implemented in fixed incremental token blocks (1024 from the providers I used) instead of simply caching up to the most recent input prompt input. It results in up to 1023 additional input (cache miss) tokens per inference.
wren6991 1 hours ago [-]
Sounds a lot like running the Qwen3.5/3.6-series models at home: you need checkpoints for the recurrent state (GDN in the case of Qwen). You avoid the miss for the common case of 100% prefix match (e.g. during interleaved tool calls and thinking) by keeping an additional checkpoint for the actual last generated token. If you're a cloud provider serving many concurrent clients then you might prefer to skip that complexity and always take the 1k worst-case prefill hit.
SWA has a similar issue. Unless you keep the entire KV prefix lying around (which is not unreasonable: you retain flop + bandwidth benefits but lose capacity benefits), you need to start 1 window back from the rollback point, in order to refill the sliding window before going into normal prefill.
cubefox 1 hours ago [-]
This comment smells LLM generated.
losvedir 49 minutes ago [-]
Huh, which model? I didn't pick up any of the tells. It seemed normal enough to me. (I vouched it, oops.)
Bolwin 1 hours ago [-]
Screams it in fact
abroadwin 1 hours ago [-]
"Quietly"
cubefox 1 hours ago [-]
"at frontier scale"
nextaccountic 34 minutes ago [-]
"—"
Rendered at 19:55:22 GMT+0000 (Coordinated Universal Time) with Vercel.
Because it's RNN-like, it has an inherent idea that X comes before Y which comes before Z in the sequence XYZ. Transformers, by default, don't have that. They operate on sets, unordered collections of unique items. They have no idea where those items are in relation to each-other so you have to clue them in.
Because There are 3 KDA layers per attention layer, and 3 KDA layers before the first attention layer, every single token position is going to be able to learn information about where it is in the sequence before the first actual attention layer.
RoPE is actually a bit destructive, so being able to omit it like this is very convenient. Models like Gemma-4 have a similar structure with 5:1 Sliding Window Attention (SWA) layers for every global attention layer. These are cheap, shitty attention layers which handle local information and which go in-between the big powerful ones, KDA serves the same role in this model. In Gemma only the SWA layers have RoPE while the global attention layers omit it. SWA is actual attention, even if it only operates on a small sliding window, so it needs the positional embedding. KDA isn't, so it doesn't.
>Curious to see if it holds up at frontier scale.
I don't know how much more frontier scale you can get than this, but yes, there's no reason why that wouldn't work at larger scales. Honestly, more parameters just makes it easier for the KDA layers to communicate that positional information better.
> Seems to benchmark
yes, but in human usage the differences show up
1. https://news.ycombinator.com/item?id=49019012 (original chinese)
2. https://news.ycombinator.com/item?id=49052912 (translated english [pdf])
SWA has a similar issue. Unless you keep the entire KV prefix lying around (which is not unreasonable: you retain flop + bandwidth benefits but lose capacity benefits), you need to start 1 window back from the rollback point, in order to refill the sliding window before going into normal prefill.