Chapter 8

Direct Alignment Algorithms

Methods like DPO that optimize preference data directly instead of fitting a separate reward model first.

Foundations 1-3

SFT 4

Reward Signals 5

Policy Optimization 6-9

Preferences & Data 10-13

Controls & Product 14-17

Direct alignment removes the explicit reward-model stage

DPO and related methods ask whether we can move the policy toward preferred outputs using pairwise preference data directly.

This is appealing because it cuts out one learned component and simplifies the training stack.

The gain is simplicity. The tradeoff is that you lose some of the flexibility that comes from having an explicit reusable reward model.

Chapter 8 When DPO first released it sparked a fierce debate in the research community about how to best do RLHF and preference learning. This meme is a great job capturing the sentiment, where the debate often felt forced and over the top, but many people both getting started and in top labs were getting immense benefit out of DPO. DPO simplicity meme, credit Tom Goldstein.

DPO is still solving the same alignment problem

The objective looks different, but conceptually it still prefers chosen completions over rejected ones while anchoring the model to a reference policy.

So direct alignment is best seen as an alternate optimization route through the same preference-learning landscape.

That is why DPO often sits beside RLHF in practice rather than replacing it conceptually.

Chapter 8 Sketch of preference displacement in DPO.

Simplicity does not remove tradeoffs

DPO has numerical concerns, data sensitivity, and known pathologies such as displacement effects under some conditions.

The absence of an explicit reward model is not the same thing as the absence of proxy risk.

It simply means the proxy is baked into the preference objective itself rather than surfaced as a separate scorer.

Review

5 quick checks

Questions with revealed answers.

1 What is the core promise of direct alignment methods like DPO?

They optimize preference data directly, avoiding a separate explicit reward-model training stage.

2 Why is DPO conceptually related to RLHF even though the objective looks different?

Because both approaches are trying to make preferred completions more likely while controlling drift from a reference policy.

3 What is the main practical attraction of DPO?

It simplifies the training pipeline and can be easier to run than full reward-model-plus-RL setups.

4 What risk remains even without an explicit reward model?

The optimization can still reflect and exploit imperfections in the preference data and objective.

5 How should you compare DPO and RLHF in one sentence?

They are two different optimization paths for turning preference data into policy updates.