RLHF RLHF Chapter notes

Chapter 16

Evaluation

How to tell whether a post-training method actually helped, and why evaluation quality often lags behind optimization quality.

Foundations 1-3
SFT 4
Reward Signals 5
Policy Optimization 6-9
Preferences & Data 10-13
Controls & Product 14-17

Evaluation is more fragile than leaderboard culture suggests

Prompt formatting, few-shot choices, and chain-of-thought scaffolding can all change reported performance dramatically.

That means many benchmark comparisons are not stable measurements of underlying model quality.

An evaluation is only as good as the protocol wrapped around it.

Report from Epoch AI showing how major AI evaluations are rapidly saturated over time (saturation is when a given benchmark reaches full performance and models no longer have meaningful signal). License CC-BY.
Chapter 16 Report from Epoch AI showing how major AI evaluations are rapidly saturated over time (saturation is when a given benchmark reaches full performance and models no longer have meaningful signal). License CC-BY.

Labs rely on internal evals because they are closer to product reality

Public benchmarks are useful but saturate quickly and are often contaminated.

Strong teams therefore build internal evaluation suites tied to their actual goals, failure modes, and product use cases.

This mirrors the broader RLHF lesson: the best metric is usually the one matched to your real objective, not the most famous one.

Evaluation is the counterweight to proxy chasing

If optimization is always trying to exploit the target, evaluation is the process of updating your picture of whether the target is still meaningful.

This is why evaluation must evolve alongside training methods.

A stale benchmark is not a safety rail. It is just another proxy waiting to be gamed.

Review

5 quick checks

Questions with revealed answers.

1 Why can benchmark scores be misleading?

Because results can depend heavily on prompting choices, protocol details, contamination, and benchmark saturation.

2 Why do strong labs build internal evaluations?

Because internal evals can be tied directly to the product behaviors and failure modes they actually care about.

3 What does it mean for a benchmark to saturate?

It means models approach ceiling performance, so the benchmark stops providing useful differentiation or signal.

4 How does evaluation relate to over-optimization?

Independent evaluation is one of the main ways to detect when training improvements on the proxy are no longer genuine.

5 What is the most important habit for evaluation design?

Treat it as an evolving measurement system that must track real objectives, not as a fixed scoreboard.