Chapter 5

Reward Modeling

How human comparisons or verifier labels become a scalar signal that later training can optimize.

Foundations 1-3

SFT 4

Reward Signals 5

Policy Optimization 6-9

Preferences & Data 10-13

Controls & Product 14-17

A reward model stands in for the environment

In ordinary RL, the environment returns reward. In RLHF, we usually have to learn that reward function from data because the target behavior is subjective and open-ended.

A reward model is therefore a proxy for human judgment, not truth itself.

That distinction is the source of both RLHF’s power and its fragility.

Chapter 5 The reward model in RLHF plays the role of the environment component that returns rewards in standard RL. The key difference is that in RLHF, we get to control and learn this reward function from human preferences, rather than having it fixed by the environment.

Chapter 5 Training a preference reward model requires pairs of chosen and rejected completions. The model computes a scalar score at the end-of-sequence (EOS) token for each, and the contrastive loss depends only on the score difference between the two.

Preference reward models learn comparisons

The classic setup takes a prompt plus two completions and trains the model to assign a higher scalar score to the preferred completion.

Only the relative ordering matters for the loss, which is why Bradley-Terry style objectives dominate this stage.

The output score is useful because it compresses a fuzzy human choice into something an optimizer can repeatedly query.

Chapter 5 At inference time, an outcome reward model outputs per-token correctness probabilities. Prompt tokens are masked (e.g., label=-100), while completion tokens each receive a probability indicating whether the model believes the response leads to a correct answer.

Outcome and process reward models change the supervision granularity

Outcome reward models predict whether a final answer is correct. Process reward models score the quality of intermediate reasoning steps.

These variants matter most in reasoning-heavy tasks where a single preference score can hide where an answer went wrong.

The supervision target is the big design choice: whole completion, final correctness, or step-level reasoning.

Chapter 5 Training an outcome reward model uses offline labels from a verifier or dataset (e.g., all 1s for correct completions). Each completion token is trained with binary cross-entropy against the outcome label, and per-token probabilities are aggregated into a final score for verification, filtering, or reranking.

The reward is useful precisely because it is imperfect

A reward model can generalize human preferences across far more examples than humans could score directly.

But every error in the reward model becomes something the optimizer can exploit.

That is why later chapters pair reward models with regularization, online data refreshes, and careful evaluation.

Chapter 5 Process reward models provide supervision only at step boundaries (e.g., newline tokens). Each step receives a 3-class label: correct (+1), neutral (0), or incorrect (-1). All other tokens are masked during training.

Review

5 quick checks

Questions with revealed answers.

1 Why do RLHF systems often need a reward model at all?

Because the desired behavior is usually not given by a fixed environment reward, so we learn a proxy from preferences or correctness labels.

2 What does a standard preference reward model actually learn?

It learns to score chosen completions above rejected completions for the same prompt.

3 How is an ORM different from a PRM?

An ORM judges final-answer correctness, while a PRM gives supervision at intermediate reasoning steps.

4 Why are reward models dangerous to optimize too aggressively?

Because the optimizer can discover behaviors that increase the proxy score without improving real user value.

5 What is the most important caveat to remember when showing reward model scores?

A high reward score is evidence of matching the proxy, not proof of genuine alignment.