Chapter 7

Reasoning Training & Inference-Time Scaling

Why RL surged again in the reasoning era, and how verification-driven training differs from classic preference optimization.

Foundations 1-3

SFT 4

Reward Signals 5

Policy Optimization 6-9

Preferences & Data 10-13

Controls & Product 14-17

Reasoning models changed the reward source

For many reasoning tasks, we can often verify whether an answer is correct without asking a human to compare two free-form completions.

That makes RL much more attractive, because the reward can be cheaper, denser, and more objective than generic preference judgments.

This is the shift from classic RLHF toward RL with verifiable rewards.

Chapter 7 RLVR in the form of an RL feedback loop. Instead of a reward model, a verification function is used.

Inference-time scaling and training-time scaling feed each other

Reasoning systems often improve by generating more thoughts, more candidates, or more compute at inference time.

RL then trains the model to use that extra computation more effectively rather than wasting it.

So the training target is no longer only style or helpfulness. It is also productive use of deliberation.

The reward loop is cleaner but still incomplete

Verification helps because correctness is less ambiguous than preference. But it does not remove all alignment concerns.

The model can still overfit to the verifier, exploit surface-level shortcuts, or become strong at narrow measurable tasks without becoming broadly reliable.

Reasoning RL is simpler in some ways, not solved in all ways.

Review

5 quick checks

Questions with revealed answers.

1 Why did RL become newly attractive in reasoning models?

Because many reasoning tasks provide verifiable rewards, making optimization cleaner and cheaper than relying only on human preference labels.

2 What is the main difference between RLHF and RLVR?

RLHF usually optimizes a learned or human-derived preference signal, while RLVR optimizes a verifier-backed reward such as correctness.

3 What does inference-time scaling mean in this context?

Giving the model extra compute at generation time through more steps, more samples, or more deliberation.

4 Why does a verifier not eliminate alignment concerns?

Because the model can still exploit the verifier or become narrow in ways that do not generalize to broader user goals.

5 What is the core conceptual link between training and inference-time scaling?

Training can teach the model how to use extra inference-time compute effectively rather than treating it as random extra effort.