TL;DR
- The idea
- Compress the knowledge of a costly ensemble into one small, deployable model.
- The intuition
- A model’s tiny probabilities on the wrong classes — a BMW looks a bit like a truck, never a carrot — carry how it generalizes.
- The mechanism
- Raise the softmax temperature T so the teacher’s soft targets expose that structure; the student trains on them.
- The result
- Soft targets alone cut a small MNIST net’s errors from 146 to 74 — no other regularizer.
The wrong answers are the lesson
A trained model knows more than its top prediction — the rest of the distribution is the part we usually throw away.
A hard, one-hot label says only this is a 2. But the teacher’s full softmax output also says how much a 2 resembles a 3 or a 7 — the similarity structure over classes. That structure, an image of a BMW may only have a very small chance of being mistaken for a garbage truck, but that mistake is still many times more probable than mistaking it for a carrot, is what the paper calls dark knowledge. Because these high-entropy soft targets carry more information per case and less gradient variance than hard targets, a student can learn from far less data. Training a small model to match the teacher’s soft targets is distillation.
We can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model.
Turn up the temperature to reveal it
Drag the dial — watch the spike on “2” melt and the dark-knowledge classes emerge.
Temperature · T
1.0
P(class “2”)
100%
A hard spike: the teacher says “2” and almost nothing else. The similarity structure is hidden. · dark-knowledge mass (3 + 7): 0.3%
§2 · Eq. 1 · the temperature knob
The knob is a single scalar T inside the softmax: T is a temperature that is normally set to 1; using a higher value for T produces a softer probability distribution over classes. Train the student at the same high T and the two logits distributions are matched. In fact matching the logits of the cumbersome model is actually a special case of distillation — the high-temperature limit. But the limit is not always best: when the distilled model is much too small, intermediate temperatures work best — which suggests ignoring the large negative logits can be helpful.
Soft targets nearly halve the errors — guess it first
Then see all three MNIST endpoints side by side.
Before the reveal — how many errors did soft targets leave?
Small net on MNIST, regularized only by soft targets (T=20)
MNIST test errors · lower is better ↓
A big net hits 67 errors; the same small net trained the ordinary way, 146. Soft targets close nearly all of that gap.
§3 · Preliminary experiments on MNIST
It can learn a digit it never saw
Remove every example of one digit from the student's transfer set. It still learns the digit — from the shadows in the soft targets.
The transfer set contains
0examples of the digit 3
The student never sees a single 3. Only the teacher’s soft targets for other digits leak that a 3 exists. So — can it recognise one?
§3 · omitting a digit from the transfer set
One net, an ensemble's accuracy — in production
On a real speech recognizer, a single distilled model reaches the frame accuracy and word-error rate of a 10-model ensemble.
The distilled single model lands at 60.8% frame accuracy and matches the ensemble’s 10.7% WER — one net, the ensemble’s quality.
More than 80% of the improvement in frame classification accuracy achieved by an ensemble of 10 models is transferred to the distilled model — the same efficiency the authors saw on MNIST, now on a heavily used commercial system.
Ensembles of specialists for huge datasets
When a dataset has 15,000 classes, a full ensemble is infeasible — so train cheap specialists on the confusable subsets instead.
A specialist focused on a narrow, confusable subset of classes trains fast and in parallel — but the main problem with specialists that focus on making fine-grained distinctions is that they overfit very easily, and the paper’s fix is (again) soft targets from the generalist. On Google’s JFT dataset of 100M images, with 61 specialist models, there is a 4.4% relative improvement in test accuracy overall. A mixture of experts is harder to parallelize; a distilled specialist ensemble is not.
Soft targets are a powerful regularizer
The through-line of the whole paper: soft targets stop a model from overfitting, letting it generalize from a fraction of the data.
On the full set both regimes sit near 58.9%. Cut the data to 3% and hard targets collapse to 44.5% (overfitting), while soft targets hold 57.0% — and converge without early stopping. (The 100%-soft cell is not tabulated in the paper.)
Soft targets allow a new model to generalize well from only 3% of the training set, because they transfer the teacher’s learned notion of which classes resemble which — a far richer signal than a one-hot label for fighting overfitting.
Five things to remember
The wrong answers are the lesson.
A teacher's tiny probabilities on wrong classes encode how it generalizes — a 2 looks a bit like a 3 and a 7. Hard one-hot labels throw all of that away.
The core claim: distill an ensemble into one deployable modelDark knowledge: wrong-answer probabilities carry information
Turn up the temperature to reveal it.
Raising T in the softmax flattens the probability spike so the teacher exposes its 'dark knowledge'; the student trains at the same high T. Matching logits is the high-T limit.
Matching logits is a special case of distillationThe temperature knob on the softmaxIntermediate temperatures work best for small students
Soft targets are a powerful regularizer.
On MNIST, soft targets alone nearly halve a small net's errors (146 → 74) — and let a model generalize from just 3% of the data (57.0% vs 44.5%) without early stopping.
MNIST baseline: big net 67 errors, small unregularized 146Soft targets alone cut the small net's errors 146 -> 74Table 5 — soft targets generalize from 3% of the data
It can learn a class it never saw.
With zero 3s in the transfer set, cross-class structure in the soft targets still lets the distilled model classify 98.6% of test 3s correctly after a bias fix.
Stress test: learning a digit it never saw
It scales to real systems.
One distilled net captured over 80% of a 10-model speech ensemble's accuracy gain, and 61 cheap specialists added 4.4% relative accuracy on JFT's 15,000 classes.
Table 1 — distilled single model matches a 10-model ensemble (speech)Over 80% of the ensemble's gain is transferredThe specialist obstacle: fine-grained experts overfit61 specialists add 4.4% relative accuracy on JFT
And the honest boundary the authors leave open: we have not yet shown that we can distill the knowledge in the specialists back into the single large net.