RLHF RLHF Chapter notes

Chapter 17

Product, UX, and Model Character

Why post-training is not only about correctness or safety, but also about shaping the assistant’s persona and fit to a product.

Foundations 1-3
SFT 4
Reward Signals 5
Policy Optimization 6-9
Preferences & Data 10-13
Controls & Product 14-17

A model is experienced as a character

Users do not encounter weights or loss functions. They encounter a style of interaction that feels more or less helpful, warm, cautious, or annoying.

This is why character training is a product problem as much as a research problem.

Post-training decides not only what the model can do, but how it feels to use.

The persona vector extraction and intervention pipeline. Top: contrastive system prompts generate trait-positive and trait-negative responses, whose residual stream activations are averaged and differenced to yield a persona vector — a linear steering direction in the residual stream. Bottom: at inference time, the persona vector is subtracted from the residual stream at selected layers, suppressing the trait and shifting the model's output toward the desired behavior. Adapted from Chen et al. (2025).
Chapter 17 The persona vector extraction and intervention pipeline. Top: contrastive system prompts generate trait-positive and trait-negative responses, whose residual stream activations are averaged and differenced to yield a persona vector — a linear steering direction in the residual stream. Bottom: at inference time, the persona vector is subtracted from the residual stream at selected layers, suppressing the trait and shifting the model's output toward the desired behavior. Adapted from Chen et al. (2025).

Persona is increasingly measurable and steerable

Work on persona vectors and assistant-axis style interventions suggests that stylistic behavior can be represented and manipulated more explicitly than many people expected.

That creates new power for product shaping, but also new responsibility.

If you can steer persona, then persona becomes part of your alignment surface.

(Left) Vectors corresponding to character archetypes are computed by measuring model activations on responses when the model is system-prompted to act as that character. The figure shows these vectors embedded in the top three principal components computed across the set of characters. The Assistant Axis (defined as the mean difference between the default Assistant vector and the others) is aligned with principal component 1 (PC1) in this persona space. Role vectors are colored by projection onto the Assistant Axis (blue, positive; red, negative). Results from Llama 3.3 70B are pictured here. (Right) In a conversation between Llama 3.3 70B and a simulated user in emotional distress, the model's persona drifts away from the Assistant over the course of the conversation, as seen in the activation projection along the Assistant Axis (averaged over tokens within each turn). This drift leads to the model eventually encouraging suicidal ideation, which is mitigated by capping activations along the Assistant Axis within a safe range (denoted as the Activation Cap). From Lu et al. [-@lu2026assistant], licensed under CC BY 4.0.
Chapter 17 (Left) Vectors corresponding to character archetypes are computed by measuring model activations on responses when the model is system-prompted to act as that character. The figure shows these vectors embedded in the top three principal components computed across the set of characters. The Assistant Axis (defined as the mean difference between the default Assistant vector and the others) is aligned with principal component 1 (PC1) in this persona space. Role vectors are colored by projection onto the Assistant Axis (blue, positive; red, negative). Results from Llama 3.3 70B are pictured here. (Right) In a conversation between Llama 3.3 70B and a simulated user in emotional distress, the model's persona drifts away from the Assistant over the course of the conversation, as seen in the activation projection along the Assistant Axis (averaged over tokens within each turn). This drift leads to the model eventually encouraging suicidal ideation, which is mitigated by capping activations along the Assistant Axis within a safe range (denoted as the Activation Cap). From Lu et al. [-@lu2026assistant], licensed under CC BY 4.0.

The product loop closes the circle

The real destination of RLHF is not a leaderboard win. It is a model that behaves well in a product with real users and real feedback loops.

So the entire pipeline should be read backward from UX: what behaviors matter, how will users experience them, and what data will reflect that?

This is the chapter that turns technical alignment back into product design.

Review

5 quick checks

Questions with revealed answers.

1 Why does model character matter in post-training?

Because users experience the assistant through tone, demeanor, and interaction style, not only through task success.

2 What do persona-vector style methods suggest?

That some aspects of model behavior and persona can be represented and steered more explicitly than simple prompt wording alone.

3 Why is persona an alignment issue and not only a UX issue?

Because style changes what advice is given, how safety boundaries are expressed, and how users interpret the model’s behavior.

4 How should product concerns influence the whole RLHF pipeline?

They should define which behaviors matter, which data to collect, and which evaluations reflect real user experience.

5 What is the final takeaway of this chapter?

RLHF is ultimately about shaping model behavior in lived product contexts, not only optimizing abstract training objectives.