Chapter 15

Regularization

The family of constraints that keep optimization productive instead of letting it destroy useful prior behavior.

Foundations 1-3

SFT 4

Reward Signals 5

Policy Optimization 6-9

Preferences & Data 10-13

Controls & Product 14-17

Regularization says what should not change too fast

In RLHF, unconstrained optimization often damages broad capabilities or pushes the model into brittle corners of behavior space.

Regularization gives the training process an anchor by penalizing updates that move too far from a useful prior policy.

This is why KL terms show up so often across RL and direct alignment methods.

Chapter 15 Forgetting dynamics for forward KL (SFT) versus reverse KL (RL). The "old" mode represents prior knowledge, the "new" mode represents the target task. Forward KL stretches the policy to cover the target and pulls mass away from the old mode (top right), while reverse KL shifts the new mode toward the target without disturbing the old mode (bottom right). From Chen et al. 2025, with permission of the author.

KL is the central language of restraint

Forward and reverse KL induce different pressures, and those pressures show up in forgetting, mode coverage, and training dynamics.

The important intuition is not the exact formula first. It is the behavioral question: do you want to preserve old modes, cover new ones, or bias toward minimal change?

That choice shapes what knowledge survives post-training.

Chapter 15 Bias toward KL-minimal solutions reduces forgetting. (Left) Among policies that solve the new task, RL converges to those closest in KL to the base model. (Right) This KL bias yields higher prior-task retention at matched new-task performance compared to SFT. From Shenfeld, Pari, and Agrawal 2026. License CC-BY.

Regularization is about retention as much as safety

Much of the value of regularization is that it protects capabilities the base model already had.

In the post-training era, teams care not only about making a model more helpful, but also about not erasing general competence.

So regularization is both a stability tool and a memory-preservation tool.

Review

5 quick checks

Questions with revealed answers.

1 Why is regularization so central in RLHF?

Because it limits destructive policy drift while the model is being pushed toward a narrower post-training objective.

2 What practical job does KL often do?

It keeps the updated policy close to a reference model so useful prior behavior is not discarded too quickly.

3 Why do different KL choices matter?

Because they bias the model toward different tradeoffs in coverage, conservatism, and forgetting.

4 Why is regularization about retention, not only safety?

Because post-training should improve target behavior without unnecessarily erasing the base model’s broader capabilities.

5 What is the cleanest intuition for regularization in one line?

Optimization tells the model what to improve; regularization tells it what not to abandon.