RLHF
Reinforcement Learning from Human Feedback
Concise chapter notes with the original figures and short review questions.
Chapters 1-3
Introduction to RLHF
A compact orientation to what RLHF changes, why the field formed, and how the modern training loop fits together.
Open chapterChapter 4
Instruction Fine-tuning
The step that turns a base next-token model into something that reliably participates in the question-answer format.
Open chapterChapter 5
Reward Modeling
How human comparisons or verifier labels become a scalar signal that later training can optimize.
Open chapterChapter 6
Reinforcement Learning (Policy Gradients)
The main optimization family for pushing a model toward higher reward while trying not to let it drift too far from useful behavior.
Open chapterChapter 7
Reasoning Training & Inference-Time Scaling
Why RL surged again in the reasoning era, and how verification-driven training differs from classic preference optimization.
Open chapterChapter 8
Direct Alignment Algorithms
Methods like DPO that optimize preference data directly instead of fitting a separate reward model first.
Open chapterChapter 9
Rejection Sampling
A simple but powerful recipe: generate many candidates, score them, keep the best, and fine-tune on the survivors.
Open chapterChapter 10
The Nature of Preferences
The conceptual chapter: what a preference really is, why RLHF needs one, and why the target is philosophically messier than it first sounds.
Open chapterChapter 11
Preference Data
How preference labels are actually gathered, what interfaces shape the labels, and why the collection process itself introduces bias.
Open chapterChapter 12
Synthetic Data & Distillation
How models increasingly learn from other models, and why synthetic feedback now sits at the center of many post-training pipelines.
Open chapterChapter 13
Tool Use & Function Calling
How assistant behavior expands once the model can call external tools instead of only emitting plain language.
Open chapterChapter 14
Over Optimization
What it looks like when the model gets better at the proxy objective faster than it gets better at what people actually want.
Open chapterChapter 15
Regularization
The family of constraints that keep optimization productive instead of letting it destroy useful prior behavior.
Open chapterChapter 16
Evaluation
How to tell whether a post-training method actually helped, and why evaluation quality often lags behind optimization quality.
Open chapterChapter 17
Product, UX, and Model Character
Why post-training is not only about correctness or safety, but also about shaping the assistant’s persona and fit to a product.
Open chapter