RLHF RLHF Chapter notes

Chapters 1-3

Introduction to RLHF

A compact orientation to what RLHF changes, why the field formed, and how the modern training loop fits together.

Foundations 1-3
SFT 4
Reward Signals 5
Policy Optimization 6-9
Preferences & Data 10-13
Controls & Product 14-17

Why post-training exists

Pretraining teaches broad world knowledge, but it does not by itself produce the style, reliability, and responsiveness people expect from assistants.

RLHF is the family of post-training methods that pushes a base model toward useful behavior by turning vague human judgments into a training signal.

The core mental move is simple: instead of asking only what token comes next, we also ask which completed answer is more helpful.

A rendition of the early, three stage RLHF process with SFT, a reward model, and then optimization.
Chapter 1 A rendition of the early, three stage RLHF process with SFT, a reward model, and then optimization.
Timeline of key developments in RLHF discussed in this chapter, from early work on RL from preferences through the adoption of RLHF in large language models.
Chapter 2 Timeline of key developments in RLHF discussed in this chapter, from early work on RL from preferences through the adoption of RLHF in large language models.
The core RLHF loop from Christiano et al. (2017): the reward predictor is trained asynchronously from comparisons of trajectory segments, and the agent maximizes predicted reward.
Chapter 2 The core RLHF loop from Christiano et al. (2017): the reward predictor is trained asynchronously from comparisons of trajectory segments, and the agent maximizes predicted reward.

The field came from preference learning

Modern RLHF did not appear fully formed with chatbots. It grew out of earlier work on learning reward functions from human comparisons in ordinary RL.

The breakthrough in language models was realizing that the same comparison loop could supervise open-ended text, not only actions in a simulator.

That history matters because it explains why reward models, KL penalties, and asynchronous feedback pipelines still show up today.

Standard RL loop
Chapter 3 Standard RL loop
Each term in the trajectory distribution (@eq:rl_dynam) mapped to the thermostat RL example.
Chapter 3 Each term in the trajectory distribution (@eq:rl_dynam) mapped to the thermostat RL example.
CartPole environment showing state variables ($x$, $\dot{x}$, $\theta$, $\dot{\theta}$) and actions ($\pm F$).
Chapter 3 CartPole environment showing state variables (xx, x˙\dot{x}, θ\theta, θ˙\dot{\theta}) and actions (±F\pm F).

The pipeline is not one algorithm

RLHF is a pipeline of stages: instruction tuning to get the model into the right interaction format, data collection to define good behavior, a reward signal to score generations, and an optimizer to push the policy toward those scores.

Different labs swap pieces in and out. Some use policy gradients, some use direct preference optimization, some rely heavily on rejection sampling or synthetic feedback.

The stable concept is not the exact recipe. It is the idea of translating human preferences into an objective that can shape model behavior.

Standard RLHF loop
Chapter 3 Standard RLHF loop
A rendition of the early, three stage RLHF process with SFT, a reward model, and then optimization.
Chapter 3 A rendition of the early, three stage RLHF process with SFT, a reward model, and then optimization.

Canonical recipes differ for a reason

InstructGPT, Tulu 3, and DeepSeek-style reasoning pipelines all share the same broad logic, but they emphasize different targets: assistant behavior, capability breadth, or reasoning performance.

This is why you should think in terms of tradeoffs instead of a single 'correct' RLHF stack.

Every later chapter is one lens on that same design space: what signal you optimize, where it comes from, and what you risk by trusting it.

A rendition of modern post-training with many rounds.
Chapter 3 A rendition of modern post-training with many rounds.
A summary of the Tülu 3 recipe with target skills and multi-step training recipe. Lambert et al. 2024, License CC-BY.
Chapter 3 A summary of the Tülu 3 recipe with target skills and multi-step training recipe. Lambert et al. 2024, License CC-BY.

Review

5 quick checks

Questions with revealed answers.

1 Why is pretraining alone usually not enough for a chat assistant?

Because pretraining optimizes next-token prediction on broad data, not the specific behaviors people want from an assistant such as instruction following, tone, and preference-sensitive helpfulness.

2 What is the core idea that unifies the RLHF pipeline?

Human judgments are turned into a usable training signal that can steer a language model after pretraining.

3 Why is it misleading to talk about RLHF as one fixed recipe?

Different systems use different combinations of instruction tuning, reward modeling, policy optimization, direct alignment, rejection sampling, and synthetic data. The stable part is the preference-driven post-training goal.

4 What role did earlier preference-learning work play?

It supplied the conceptual template for learning a reward proxy from comparisons and then optimizing behavior against that learned signal.

5 What should someone leave this page knowing before moving on?

That RLHF is best understood as a modular post-training pipeline for turning human preferences into better model behavior, not as a single algorithm.