TL;DR
- The gap
- Maximizing the likelihood of human-written text is not the same as producing outputs humans judge to be good.
- The idea
- Collect human preferences between pairs of summaries and train a reward model to predict the preferred one.
- The optimization
- Then train a policy with reinforcement learning (PPO) to maximize the reward model’s score.
- The result
- The 6.7B feedback model’s summaries are preferred to the human references about 65% of the time.
Likelihood is not quality
Summarizers are trained to copy human references and scored with ROUGE — but neither is what we actually want.
The standard recipe is supervised learning: maximize the likelihood of human-written reference summaries. It works, but there is a catch — both of these metrics are rough proxies for what we really care about — summary quality. Optimizing a proxy is not the same as optimizing the goal, and that gap is a problem of alignment.
There is still a misalignment between this fine-tuning objective — maximizing the likelihood of human-written text — and what we care about.
The fix the paper proposes: stop imitating references, and instead optimize directly for what humans prefer. That means turning a fuzzy human judgment into a differentiable reward — the job of a reward model.
RLHF, in three steps that repeat
Collect comparisons, train a reward model, optimize a policy against it — the loop the hero graphic runs.
Collect human comparisons
Show a labeler two summaries of the same post; they pick the better one. Repeat tens of thousands of times.
Reward model from comparisons
Train a reward model
Fit a reward model to predict which summary a human would prefer — its logit becomes a scalar reward.
Optimize policy against RM
Optimize the policy with PPO
Fine-tune the summarizer with reinforcement learning to maximize that reward, token by token — then loop back.
RL policy against the RM
All the models are GPT-3-style transformer decoders. The two headline runs use models with 1.3 billion (1.3B) and 6.7 billion (6.7B) parameters. Concretely, step three treats the reward as the RL objective: we treat the logit output of the reward model as a reward that we optimize using reinforcement learning. The full procedure — comparisons, reinforcement learning, and the policy update — is the canonical RLHF diagram.
Here is that loop as a live figure — the same cycle the hero graphic performs, now with the cited procedure attached:
Collect comparisons → train a reward model → optimize the policy — then repeat.
The full feedback loop
Do humans actually prefer it?
Predict where the 6.7B feedback model lands, then explore preference across every model size.
Before the reveal — how often did humans prefer the 6.7B feedback model to their own references?
6.7B human-feedback model vs human reference summaries (TL;DR)
It beats the references — and transfers
The feedback model doesn't just edge out the baseline; it out-scores the human-written summaries it learned from.
Summaries from our 6.7B PPO model achieve a 7/7 overall score 45% of the time — versus 20% for the supervised baseline and 23% for the human references.
A model trained only on Reddit generalizes to CNN/DM news: our 6.7B TL;DR human feedback model nearly matches T5 despite never being trained to summarize news articles.
The reward model is not the true goal
Push the policy too hard against the reward model and real preference peaks, then collapses. Drag the dial.
This is Goodhart’s law in action: as we optimize further, true preferences fall off compared to the prediction, and eventually the reward model becomes anti-correlated with human preferences. Reassuringly, the reward model is still sensitive to meaning — on role-reversed edits our models reliably select the original summary (92.9% of the time for 1.3B, 97.2% for 6.7B).
ROUGE stops tracking quality
On the better model's outputs, the classic metric falls to chance while the learned reward model stays predictive.
Best-of-N reranking: which scorer keeps winning?
N = 128
Rerank N samples by each scorer and keep the top pick. Reward models keep improving; ROUGE peaks early and decays.
Curves are schematic, anchored to the paper’s finding that ROUGE peaks “both sooner and at a substantially lower preference rate.” §4.4 · Figure 7
While ROUGE has ~57% agreement with labelers when comparing samples from our supervised baseline models, this drops to ~50% for samples from our human feedback model — i.e. to a coin flip. The reward model is a far better optimization target than any hand-designed metric.
More data, or a bigger model?
Both sharpen the reward model — toggle to see which lever pulls harder.
What sharpens the reward model?
validation-accuracy gain from doubling the model size
Model size is the slightly stronger lever (+1.8% vs +1.1%), and the 6.7B reward model climbs toward the 66.9% inter-labeler agreement ceiling — the point where it agrees with people as often as people agree with each other. (current lever: +1.8%, the other: +1.1%)
§4.3 · Figure 6
Powerful, but expensive
Human feedback is a heavy dependency — in both compute and labeling.
Fine-tuning our 6.7B model with RL required approximately 320 GPU-days, on top of a large and costly human-labeling effort. That expense is exactly why the method has to be sample-efficient with human comparisons — and why the reward-model scaling laws above matter so much.
Five things to remember
Likelihood is not quality.
Maximizing the likelihood of human-written text — and scoring with ROUGE — only approximates what people actually want. That gap is the whole motivation.
The core misalignmentROUGE is a rough proxy
The RLHF loop, in three steps.
Collect human which-is-better comparisons, train a reward model to predict them, then optimize a policy against that reward with PPO — and repeat.
Reward model from comparisonsRL policy against the RMOptimize policy against RMThe full feedback loop
It beats the human references.
A 1.3B feedback model outdoes a 10× larger supervised model; the 6.7B feedback model is preferred to human references ~65% of the time and earns a perfect 7/7 score 45% of the time.
Beats human referencesPerfect scores 45% of timeTransfers to news, matches T5
The reward model is not the true goal.
Push the policy too hard against the RM and real human preference peaks, then collapses — the model over-optimizes a proxy (Goodhart). ROUGE, meanwhile, stops tracking quality entirely.
Over-optimizing the RM backfiresROUGE stops tracking qualityRM catches role-reversed edits
Powerful, but expensive.
Both more comparison data (~1.1%) and bigger models (~1.8%) sharpen the reward model — but RL-fine-tuning the 6.7B model cost ~320 GPU-days plus a large human-labeling effort.
Data vs model-size levers320 GPU-days to train1.3B and 6.7B models