RLHF RLHF Chapter notes

Chapter 14

Over Optimization

What it looks like when the model gets better at the proxy objective faster than it gets better at what people actually want.

Foundations 1-3
SFT 4
Reward Signals 5
Policy Optimization 6-9
Preferences & Data 10-13
Controls & Product 14-17

Over-optimization is the default failure mode of proxy training

If you optimize a learned or simplified objective hard enough, the model will eventually find ways to win on the metric without truly solving the underlying task.

This is not an edge case. It is the expected behavior of strong optimizers acting on incomplete proxies.

RLHF inherits this problem because reward models and automated judges are always approximations.

Over-optimization of an RL training run vs. downstream evaluations. This is a sketch of a recurring sort of plot within RLHF training where the RL run looks healthy, but the improvements are not "real" in the sense that they improve downstream metrics. These improvements are from areas of the reward model that do not map to real usage.
Chapter 14 Over-optimization of an RL training run vs. downstream evaluations. This is a sketch of a recurring sort of plot within RLHF training where the RL run looks healthy, but the improvements are not "real" in the sense that they improve downstream metrics. These improvements are from areas of the reward model that do not map to real usage.

Qualitative and quantitative failures both matter

Some failures look like strange tone, sycophancy, or over-refusal. Others show up as a widening gap between training reward and downstream evaluation.

The important teaching point is that not all failures are visible from the training dashboard.

A run can look healthy while actually drifting away from the behavior you care about.

Over-optimization with a train and test RM from Bai et al. 2022. License CC-BY.
Chapter 14 Over-optimization with a train and test RM from Bai et al. 2022. License CC-BY.

The problem is structural, not merely bad tuning

Better hyperparameters can delay over-optimization, but they do not remove the fundamental mismatch between proxy and goal.

That is why mature systems pair optimization with refreshes to the data, the reward model, and the evaluation suite.

You are managing an adversarial relationship between optimizer and proxy.

Review

5 quick checks

Questions with revealed answers.

1 What is over-optimization in one sentence?

It is when the model improves the training proxy more than it improves the real objective you actually care about.

2 Why is over-optimization especially relevant in RLHF?

Because RLHF often optimizes learned or simplified reward proxies rather than the full human objective.

3 What is one qualitative sign of over-optimization?

Behavior such as excessive hedging, over-refusal, or other unnatural patterns that seem good to the proxy but bad to users.

4 What is one quantitative sign?

Training reward continues to rise while independent evaluations stop improving or start getting worse.

5 Why can’t hyperparameter tuning fully solve it?

Because the root issue is mismatch between the proxy and the real target, not only optimization aggressiveness.