Written by Jakub Rusinowski · Published August 25, 2026
Moonshot AI's Kimi Linear is the first architecture to beat full attention under a controlled comparison — while cutting KV cache 75% and decoding 6.3x faster at 1M tokens. Here's how Kimi Delta Attention actually works, transcribed from the open-source kernel: the state-update equation, why per-channel gating was the unlock, where the 75% comes from arithmetically, and what the paper does not prove.
Every transformer you have ever run pays the same tax. To generate token number 100,001, the model re-reads all 100,000 tokens that came before it. It keeps them in a structure called the KV cache, and that cache grows linearly with your context — every token you add makes every future token more expensive to produce. For years the fix was obvious and unusable. Linear attention replaces the growing cache with a fixed-size memory: instead of remembering every token, keep a compressed summary and update it as you go. Constant memory, constant cost per token, no matter how long the context. The c…