Chapter 4
Instruction Fine-tuning
The step that turns a base next-token model into something that reliably participates in the question-answer format.
Message stack
Instruction tuning as a message stack
System, user, and assistant roles define the conversation format that supervised fine-tuning teaches the model to follow.
Define high-level behavior, policy, and guardrails.
Provide the task, context, or question that the assistant must answer.
Produce the demonstration target that SFT trains the model to imitate.
SFT establishes the interaction contract
Instruction tuning is the first behavioral reshape after pretraining. It teaches the model what a prompt-response exchange looks like.
Without this stage, later preference optimization has to fix too many things at once: formatting, tone, obedience, and task completion.
The practical intuition is that SFT makes the model legible before alignment methods try to make it preferable.
Chat templates are policy structure
A chat model is not just a base model with nicer data. The message roles, separators, and system-user-assistant ordering all become part of the policy interface.
If the template is inconsistent, training quality looks worse than it really is because the model is learning a broken protocol.
This is why template hygiene is a first-order concern, not a formatting afterthought.
Good SFT data teaches priorities
Instruction tuning does not only teach answer style. It teaches what kinds of requests deserve direct answers, caveats, or refusals.
The model absorbs latent preferences from the demonstrations long before an explicit reward model sees anything.
That means SFT already contains alignment decisions, even if people talk about it as a neutral stage.
Why this stage still matters later
A strong SFT model makes every later method easier: preference data becomes cleaner, reward models are easier to train, and RL spends less time repairing basic behavior.
Many systems get most of their user-facing quality from high-quality supervised fine-tuning plus careful data curation.
Think of RLHF as an amplifier. If the SFT foundation is weak, the amplifier mostly magnifies noise.
Review
5 quick checks
Questions with revealed answers.
1 What is the main job of instruction fine-tuning?
To teach a pretrained model how to behave as an instruction-following assistant in the prompt-response format.
2 Why are chat templates more than formatting details?
Because they define the structure of the policy’s inputs and outputs. If that structure is inconsistent, training teaches the wrong interaction pattern.
3 Why can SFT already be considered part of alignment?
Because the demonstrations encode preferences about tone, helpfulness, refusals, and what kinds of answers are considered acceptable.
4 How does strong SFT help later RLHF stages?
It reduces behavioral chaos, making preference collection, reward modeling, and optimization target refinement much easier.
5 What is the simplest mental model for the relationship between SFT and RLHF?
SFT gives the model the right interface and baseline behavior; later alignment methods refine which outputs are preferred within that interface.