Chapter 9
Rejection Sampling
A simple but powerful recipe: generate many candidates, score them, keep the best, and fine-tune on the survivors.
The method is simple because the idea is simple
Rejection sampling sidesteps online RL by moving the selection step into data construction.
You sample multiple completions, score them with some signal, and keep the subset you want the model to imitate.
This often gets strong results because it turns alignment into supervised learning over filtered outputs.
Selection policy matters more than it first appears
The practical difficulty is not generating candidates. It is deciding how to score and select them.
Top-per-prompt selection, pair formation, and global ranking all induce different biases in the retained data.
That means rejection sampling is really a data design method disguised as a simple filtering step.
It is often a baseline because it is robust
Compared with online RL, rejection sampling is easier to reason about, easier to debug, and often easier to reproduce.
Its weakness is that it only improves within the support of the samples you already generated.
So it is powerful when your candidate pool is good, and limited when your base model never proposes the right behavior.
Review
5 quick checks
Questions with revealed answers.
1 What are the three core steps in rejection sampling?
Generate candidate completions, score/select the ones you want, then fine-tune on those retained examples.
2 Why is rejection sampling often easier than online RL?
Because it avoids unstable policy updates during data collection and reduces the process to filtering plus supervised training.
3 What is the hidden complexity in rejection sampling?
The scoring and selection policy strongly shape what ends up in the training data.
4 What is the main limitation of rejection sampling?
It can only choose among completions the model already generated, so it cannot easily learn behaviors outside that sampled support.
5 Why is this method important in the RLHF toolbox?
Because it often delivers strong alignment improvements with much less instability and engineering complexity than full RL.