TL;DR
- The idea
- A foundation architecture that hits training parallelism, low-cost inference, and good performance simultaneously.
- The trick
- Retention has a dual form — train it as a parallel matrix op, run it as an O(1) recurrence.
- The payoff
- A 7B model decodes 8.4× faster and saves 70% of memory versus a Transformer with a KV cache.
- The claim
- The authors position RetNet as an ideal successor to Transformers for large language models.
The impossible triangle
Training parallelism, cheap inference, strong performance — pick two. Every efficient Transformer alternative gives one up.
A Transformer trains in parallel and performs well, but decoding costs grow with sequence length because of the KV cacheKV cacheThe stored keys and values a Transformer keeps for every past token during decoding; it grows with sequence length. RetNet replaces it with a fixed-size state.Full definition →. Linear attention and recurrent neural network variants make inference cheap, but sacrifice parallel training or quality. None of the previous work can break through the impossible triangle, so none has been a clear winner against Transformers.
RetNet is a foundation architecture simultaneously achieving training parallelism, low-cost inference, and good performance.
Retention: attention you can rewind
Replace softmax attention with retention — a decay-weighted mixture that can be written two mathematically equivalent ways.
RetentionretentionRetNet's replacement for attention — a decay-weighted mixture of past tokens that admits both a parallel and an O(1) recurrent form.Full definition → keeps a running, decay-weighted summary of the past. Its key property is a dual form of recurrence and parallelism: train it as one big parallel matrix multiply, then run inference as a cheap recurrence that carries a fixed-size state forward. Same weights, same outputs — two shapes.
One mechanism, three shapes
The same retention runs three ways — flip between them and watch the complexity readout change.
The retention mechanism supports three computation paradigms — parallel for fast training, recurrent for O(1) inference, and chunkwise recurrentchunkwise recurrentCompute retention in parallel inside each chunk, then carry a small state across chunks — parallel enough to train fast, cheap enough for long sequences.Full definition → to train efficiently on very long sequences.
Retention(X) = (QKᵀ ⊙ D) V
O(N²) compute · fully parallelizable
Three computation paradigms
Multi-scale decay: drag a head
Retention's memory is controlled by a per-head decay γ. Different heads remember over different horizons.
Head · γ
0.996094
Half-life
~177tokens
Each head gets its own decay rate γ = 1 − 2^(−5−h). Low heads forget fast (local detail); high heads remember far back — that spread is multi-scale decaymulti-scale decayEach head uses a different decay rate γ, so some heads remember far back and others focus locally — the ablated ingredient that lifts quality.Full definition →.
§3.6 · Ablation Studies
The only row that wins on every axis
Table 1 lines RetNet up against six alternatives across the four axes that matter.
| Model | Train parallel | Inference | Low memory | Performance |
|---|---|---|---|---|
| Transformer | ✓ | O(N) | ✓ | strong |
| Linear Transformer | ✓ | O(1) | ✕ | poor |
| Recurrent NN | ✕ | O(1) | ✕ | poor |
| RWKV | ✕ | O(1) | ✓ | ok |
| H3/S4 | ✓ | O(1) | ✕ | ok |
| Hyena | ✓ | O(N) | ✓ | ok |
| RetNet | ✓ | O(1) | ✓ | strong |
RetNet achieves training parallelization, constant inference cost, and linear long-sequence memory — the only row that wins on every axis.
Table 1 · Architecture comparison
Inference stops scaling with length
A Transformer's KV cache grows every step. RetNet's fixed state does not — so cost goes flat.
Cost vs sequence length (7B model)
RetNet’s decoding latency keeps almost the same across different batch sizes and input lengths — flat because there is no growing KV cache. Curves are schematic, anchored to the reported 7B / 8k numbers.
Figure 6 · Inference cost
Because activations don’t grow with context, RetNet’s additional memory is almost negligible (about 3%) while model weights occupy 97%. The gains show up in training too: RetNet achieves 25-50% memory saving and 7× acceleration over the standard Transformer.
Efficiency without giving up quality
The catch with efficient alternatives is usually accuracy. RetNet keeps it — and pulls ahead at scale.
At scale, it wins
RetNet tends to outperform Transformer when the model size is larger than 2B on validation perplexity.
Downstream tasks match
RetNet achieves comparable performance with Transformer on zero-shot and in-context learning settings.
Against efficient variants like RWKV, H3, and Hyena, RetNet outperforms other architectures on the in-domain set and various out-of-domain corpora. One caveat worth reading: the authors implement RetNet in vanilla PyTorch, leaving kernel fusion for future work — so the reported speedups are pre-optimization.
Five things to remember
One architecture, all three corners.
RetNet is the first design to hit training parallelism, O(1) inference, and Transformer-level quality at once — the previously impossible triangle.
Three properties at oncePrior work can't break the triangleImpossible triangle made possible
One retention, three computation shapes.
The same math is written three ways: parallel for training, O(1) recurrent for inference, and chunkwise recurrent for long sequences.
Three computation paradigmsDual form of recurrence and parallelismHow training uses both forms
Inference cost stops scaling with length.
8.4× faster decoding, 70% less memory, ~3% overhead, and latency invariant to batch size and input length — because there is no growing KV cache.
8.4× faster, 70% less memory~3% memory overheadBatch- and length-invariant latency
Efficiency without giving up quality.
RetNet matches Transformer on zero/few-shot tasks, beats efficient variants across domains, and pulls ahead above 2B parameters. Multi-scale decay is load-bearing.
Outperforms above 2B paramsComparable downstream accuracyBeats efficient variantsMulti-scale decay matters
Positioned as a Transformer successor.
The authors close by framing RetNet as an ideal successor to Transformers — with kernel-fusion speedups still left on the table as future work.
An ideal successorVanilla PyTorch, kernels left for futureModel comparison table