RLHF RLHF Chapter notes

Chapter 12

Synthetic Data & Distillation

How models increasingly learn from other models, and why synthetic feedback now sits at the center of many post-training pipelines.

Foundations 1-3
SFT 4
Reward Signals 5
Policy Optimization 6-9
Preferences & Data 10-13
Controls & Product 14-17

Synthetic data scales judgment

Human preference data is expensive and slow. Synthetic data lets a stronger model generate demonstrations, critiques, or comparisons at much larger scale.

That shift changes the economics of post-training and explains why modern pipelines can move quickly.

The core question becomes when model-generated supervision is good enough to trust.

Traditional knowledge distillation trains a smaller student model to match the soft probability distribution of a larger teacher model using KL divergence loss. Both models process the same input simultaneously, and temperature scaling ($\tau > 1$) softens the distributions to reveal more information about class relationships.
Chapter 12 Traditional knowledge distillation trains a smaller student model to match the soft probability distribution of a larger teacher model using KL divergence loss. Both models process the same input simultaneously, and temperature scaling (τ>1\tau > 1) softens the distributions to reveal more information about class relationships.

Distillation is capability transfer, not just compression

Distillation often means using a stronger teacher model to produce targets that a smaller or cheaper model can imitate.

In post-training, this can include demonstrations, synthetic preferences, rubric-based judgments, or constitution-style critiques.

So distillation is best viewed as a way of transferring behavior and judgment, not only logits.

Synthetic data generation in LLM post-training: prompts are passed through a strong model to generate completions, which are paired to create a training dataset. This dataset is then used to fine-tune smaller models via standard supervised learning. More complex pipelines may involve multiple models editing completions, generating preference pairs, or filtering for quality.
Chapter 12 Synthetic data generation in LLM post-training: prompts are passed through a strong model to generate completions, which are paired to create a training dataset. This dataset is then used to fine-tune smaller models via standard supervised learning. More complex pipelines may involve multiple models editing completions, generating preference pairs, or filtering for quality.

Synthetic feedback is powerful because it is structured

Rubrics, constitutions, and explicit judge prompts make synthetic supervision more legible and easier to iterate on.

But every rubric imports the biases of the judge model and the prompt writer.

The bottleneck shifts from raw data volume to the quality of the synthetic teacher.

Review

5 quick checks

Questions with revealed answers.

1 Why is synthetic data so important in modern post-training?

Because it provides much more scalable supervision than relying only on expensive human labels.

2 What does distillation transfer in this setting?

Not just raw predictive skill, but behaviors, judgments, preferences, and reasoning patterns from a stronger teacher.

3 Why are rubrics useful in synthetic feedback pipelines?

They make the evaluation criteria more explicit and can improve consistency in model-generated judgments.

4 What new risk comes with relying on synthetic judges?

The pipeline can inherit the biases, blind spots, and prompt sensitivities of the judge model.

5 What is the central design question for synthetic post-training?

How much of the desired behavior can be safely delegated to model-generated supervision without drifting away from real user preferences.