Improving Language Understanding by Generative Pre-Training — Breakdown

The whole paper in 20 seconds

TL;DR

The idea: Pre-train a language model on unlabeled text, then fine-tune it on each specific task.
Why it matters: Raw text is abundant; the labeled data these tasks need is scarce and expensive.
The result: One task-agnostic model beat bespoke architectures on 9 of the 12 tasks studied.
The legacy: A single task-agnostic model from generative pre-training — the template every GPT inherits.

Why it exists

Labels are the bottleneck, not text

Raw text is everywhere; the labels that supervised models need are the expensive part.

Deep NLP models are data-hungry, but labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately. The blocker was evidence, not ideas: it was unclear which optimization objective yields text representations useful for transfer. A pre-trained language model answers both questions at once — it gives an objective and a path to transfer learning.

Generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task.

The recipe

Two stages, one backbone

Learn from raw text once; adapt cheaply, many times.

First, a language-modeling objective on the unlabeled data learns the initial parameters of the network; then a supervised objective adapts those parameters to each task. Only the second stage spends labels, and it reuses the same weights — a single pre-train feeding many cheap fine-tuning runs.

The model

A 12-layer decoder-only Transformer

No recurrence — masked self-attention plus feed-forward layers, all the way down.

The backbone applies a multi-headed self-attention operation over the input context tokens, followed by position-wise feedforward layers. Concretely it is a 12-layer decoder-only Transformer with masked self-attention (768-dimensional states and 12 attention heads). It learns on over 7,000 unique unpublished books, whose long contiguous passages teach long-range structure — and the pre-training itself is strong: a very low token-level perplexity of 18.4 on the corpus. The building blocks — self-attention, feed-forward layers, token embeddings, perplexity — are the same the Transformer introduced, wired as a decoder.

The trick

Change the input, not the architecture

Structured tasks are rewritten into one delimited token sequence the frozen model already understands.

Rather than bolt task-specific modules onto the model, GPT uses traversal-style transforms that convert structured inputs into an ordered sequence the pre-trained model can process. All structured inputs become token sequences processed by the pre-trained model, followed by a linear+softmax layer. Pick a task and watch the raw input become chips:

A premise and a hypothesis, joined by a delimiter.

Start→Premise→Delim ($)→Hypothesis→Extract→entail

Every task threads the same frozen backbone plus one linear+softmax head — only the token layout changes.

§3.3 · Input TransformationsFigure 1 · caption

Here is the paper’s Figure 1 (left) — the decoder-only block itself — as an on-theme recreation:

Left: a 12× Transformer decoder block (masked self-attention, layer norm, feed forward) topped by text-prediction and task-classifier heads. Right: input transformations converting classification, entailment, similarity, and multiple-choice tasks into token sequences fed through the same Transformer and a linear layer. — Figure 1: (left) Transformer architecture and training objectives used in this work. (right) Input transformations for fine-tuning on different tasks.

The payoff

New state of the art on 9 of 12 tasks

One task-agnostic model beats architectures built for each benchmark — with a few honest misses.

The single model lands absolute improvements of 8.9% on Stories Cloze, 5.7% on RACE, and 1.5% on MultiNLI. The biggest wins are where context runs long — up to 8.9% on Story Cloze and 5.7% overall on RACE — exactly the regime a Transformer handles better than a recurrent net.

Finetuned Transformer LM vs prior best

9 / 12 tasks

MNLI · accuracyprev 80.6↑ 82.1

SNLI · accuracyprev 89.3↑ 89.9

Story Cloze · accuracyprev 77.6↑ 86.5

RACE · accuracyprev 53.3↑ 59.0

CoLA · Matthewsprev 35.0↑ 45.4

GLUE · avgprev 68.9↑ 72.8

RTE · accuracyprev 61.7↓ 56.0

The one clear miss: on the small RTE set (2,490 examples) accuracy is 56%, below the 61.7 of a multi-task biLSTM.

Abstract · 9 of 12Table 2 · Results

Why it works · 1

Every transferred layer helps

Transfer more of the pre-trained stack and accuracy keeps climbing — drag to see it.

Layers transferred

81.4dev accuracy↑ +8.9 vs 0 layers

Each transformer layer provides further benefits, up to 9% for full transfer on MultiNLI.

Figure 2 · layers transferred

The paper’s own Figure 2 — the transfer curve (left) alongside zero-shot behavior growing with pre-training (right):

Left: line chart where RACE and MultiNLI dev/train accuracy rise steadily as more pre-trained layers are transferred. Right: line chart where zero-shot task performance (sentiment, winograd, linguistic acceptability, question answering) improves with pre-training updates, Transformer beating LSTM. — Figure 2: (left) Effect of transferring increasing number of layers from the pre-trained language model on RACE and MultiNLI. (right) Zero-shot performance on different tasks as a function of LM pre-training updates.

Why it works · 2

Pre-training is what carries the gains

Guess the cost of removing it, then see the cliff.

Strip out pre-training entirely — where does the average score land?

Transformer w/o pre-training vs full model

67.5avg score

55full model 74.780

Average score across all tasks (Table 5)

74.7Full model (w/ aux LM)

75.0Transformer w/o aux LM+0.3

69.1LSTM w/ aux LM-5.6

59.9Transformer w/o pre-training-14.8

The LSTM shows higher variance in zero-shot performance — the Transformer’s inductive bias assists in transfer.

Table 5 · Model ablations

So what

Five things to remember

Labels are the bottleneck, not text.

Raw text is abundant while task labels are scarce and expensive — so learn from unlabeled books first, then spend labels only on a light adaptation step.

The motivating bottleneckWhy prior semi-supervised work stalled

One pre-train, many cheap fine-tunes.

Generative language-model pre-training on BooksCorpus, then discriminative fine-tuning on each task — a two-stage loop over a single Transformer-decoder backbone.

The recipe in one sentenceTwo-stage training procedureArchitecture: multi-layer Transformer decoderPre-training corpus: BooksCorpus

Change the input, not the architecture.

Structured inputs are rewritten into one delimited token sequence (Start · $ · Extract) so entailment, similarity, and QA all reuse the same frozen model plus a linear+softmax head.

Traversal-style input transformationsFigure 1 - inputs to token sequences

New state of the art on 9 of 12 tasks.

One task-agnostic model beats bespoke architectures — +8.9% Story Cloze, +5.7% RACE, +1.5% MultiNLI — losing clearly only on the tiny RTE set (56.0 vs 61.7).

Headline resultAbsolute improvementsLong-context QA gainsWhere it loses: RTE

Pre-training is what carries the gains.

Ablations show removing pre-training costs 14.8% across all tasks, each transferred layer adds up to 9% on MultiNLI, and the Transformer's inductive bias transfers better than an LSTM's.

Ablation - pre-training is load-bearingEvery layer helpsTransformer inductive bias helps transferConclusion

Read the full paper OpenAI writeup Back to home