BERT:

A language model is trained to predict the next token from the ones before it. That is fine for generation, but it hobbles transfer learning to tasks that need the whole sentence at once — question answering, entity tagging. The major limitation is that standard language models are unidirectional, and this limits the choice of architectures that can be used during pre-training.

The masked language model randomly masks some of the tokens from the input, and the objective is to predict the original vocabulary id of the masked word based only on its context.

BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. That single design choice — bidirectional self-attentioninstead of a left-only mask — is what separates it from the prior generation of pre-trained Transformer models. The mechanism on the right of the hero shows it live: a hidden word is recovered from context on both sides.

The masked language model (a Cloze task) hides a fraction of tokens and asks the model to recover them. In all of our experiments, we mask 15% of all WordPiece tokens in each sequence at random. A twist keeps pre-training aligned with fine-tuning, where [MASK] never appears — click through the 80/10/10 rule below.

Alongside the masked LM, BERT is pre-trained to tell whether one sentence really follows another. 50% of the time B is the actual next sentence that follows A (labeled as IsNext), and 50% of the time it is a random sentence from the corpus. No annotation is required — the labels fall out of any monolingual corpus for free.

All three pre-train on text, but their connectivity differs. Critically, however, the BERT Transformer uses bidirectional self-attention, while the GPT Transformer uses constrained self-attention where every token can only attend to context to its left. The reported sizes are BERT-BASE (110M parameters) and BERT-LARGE (340M parameters), the two configurations behind every headline number.

Figure 3

Original Recreation

Critically, however, the BERT Transformer uses bidirectional self-attention, while the GPT Transformer uses constrained self-attention where every token can only attend to context to its left.

BERT is a two-phase recipe. Pre-training runs the masked LM and next-sentence tasks on unlabeled text; fine-tuning swaps in a small task-specific head and updates all parameters end-to-end. The encoder stack — and its weights — carry straight across.

Figure 1

Original Recreation

When choosing the sentences A and B for each pre-training example, 50% of the time B is the actual next sentence that follows A (labeled as IsNext), and 50% of the time it is a random sentence from the corpus (labeled as NotNext).

This is the experiment that pins the credit on two-sided attention rather than scale or data. Flip the metric to see which tasks feel the loss most.

Both BERTBASE and BERTLARGE outperform all systems on all tasks by a substantial margin, obtaining 4.5% and 7.0% respective average accuracy improvement over the prior state of the art. BERT also works without fine-tuning: concatenating the top four hidden layers as frozen features lands within 0.3 F1 of fine-tuning the whole model.

Larger models lead to a strict accuracy improvement across all four datasets, even for MRPC which only has 3,600 labeled training examples. With enough pre-training, extreme size pays off on small fine-tuning sets too — a result that was far from obvious at the time.