LoRA: Low-Rank Adaptation

The premise of adaptation is cheap once, expensive forever. Using GPT-3 175B as an example, deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. Existing shortcuts have a catch: parameter-efficient adapter layers cut the stored weights but run in series with the frozen model, so they add real latency at serving time.

The inference latency introduced by adapter layers can be significant in an online, short-sequence-length scenario.

The weight matrices in these layers typically have full-rank, yet the authors hypothesize the update ΔW picked up while adapting to a task lives in a much smaller subspace — it has a low intrinsic rank. If that is true, you never need to store or train a full d×d update; two thin matrices suffice.

LoRA replaces the update with a rank-r product: ΔW = B·A, where A projects down to r dimensions and B projects back up. We use a random Gaussian initialization for A and zero for B, so ΔW = B·A is exactly zero at the start — training begins as the untouched pretrained model and only the two skinny matrices receive gradients.

Figure 1 — The LoRA reparametrization

Original Recreation

This is the intrinsic-rank claim, made empirical. To our surprise, a rank as small as one suffices for adapting both W_q and W_v on these datasets. Drag the rank and watch the bars stay flat:

With r = 4 and only the query and value projection matrices being adapted, the checkpoint size is reduced by roughly 10,000× (from 350GB to 35MB), and there is a 25% speedup during training on GPT-3 175B compared to full fine-tuning. At deploy time B·A folds back into W — so unlike adapters, serving latency is untouched:

The paper plots this as accuracy against trainable-parameter count. Here is an on-theme recreation of that curve — LoRA stays high and flat where prefix methods sag:

Figure 2 — Accuracy vs. trainable parameters on GPT-3 175B

Original Recreation

Comparing the learned update to the pretrained weight, the authors find ΔW aligns with directions W barely uses, then scales them up — an amplification factor of about 21.5 at r=4. This suggests that the low-rank adaptation matrix potentially amplifies the important features for specific downstream tasks that were learned but not emphasized in the general pre-training model. A tiny, targeted lever on latent capacity — not a full rewrite.

The zero-latency trick works precisely because the low-rank update is absorbed into the frozen weight. But once merged, it is not straightforward to batch inputs to different tasks with different A and B in a single forward pass. A single batch can no longer mix requests belonging to different task-specific LoRA modules unless you keep the paths un-merged.