RLHF RLHF Chapter notes

Chapter 10

The Nature of Preferences

The conceptual chapter: what a preference really is, why RLHF needs one, and why the target is philosophically messier than it first sounds.

Foundations 1-3
SFT 4
Reward Signals 5
Policy Optimization 6-9
Preferences & Data 10-13
Controls & Product 14-17

Preferences are the target because utility is hard to observe directly

RLHF exists because human goals are rich, contextual, and usually not available as explicit numeric rewards.

Preference judgments are a practical stand-in: humans may struggle to define the perfect answer, but they can often compare two answers.

This is the operational bridge from abstract human values to trainable objectives.

The timeline of the integration of various subfields into the modern version of RLHF. The direct links are continuous developments of specific technologies, and the arrows indicate motivations and conceptual links.
Chapter 10 The timeline of the integration of various subfields into the modern version of RLHF. The direct links are continuous developments of specific technologies, and the arrows indicate motivations and conceptual links.

A preference is not the same as truth

Some preferences track correctness, some track style, some track safety, and some conflict with each other.

That means the learning target is plural and sometimes inconsistent by construction.

Any alignment pipeline that treats preference data as a single clean scalar objective is already simplifying reality.

RLHF is a theory of approximation

The field works by taking a messy social target and compressing it into tractable training signals.

The reason it succeeds at all is that many useful preferences are stable enough to model.

The reason it remains difficult is that the compression always leaves something out.

Review

5 quick checks

Questions with revealed answers.

1 Why are preferences so central to RLHF?

Because they provide a practical way to supervise behavior when explicit numeric objectives are unavailable or incomplete.

2 Why is a preference not the same thing as truth?

Because preferences can reflect style, values, safety norms, and context-specific tradeoffs, not only factual correctness.

3 What makes preference targets hard to optimize perfectly?

They are often plural, context-dependent, and internally inconsistent, so any single scalar proxy is a simplification.

4 Why can pairwise comparison still work despite that messiness?

Because humans are often better at choosing between alternatives than at writing down a complete objective.

5 What should viewers remember from this chapter before looking at data collection?

That the dataset is only a measurement of preferences, not the preferences themselves in full.