Chapter 11

Preference Data

How preference labels are actually gathered, what interfaces shape the labels, and why the collection process itself introduces bias.

Foundations 1-3

SFT 4

Reward Signals 5

Policy Optimization 6-9

Preferences & Data 10-13

Controls & Product 14-17

The collection interface shapes the signal

Preference data is not abstract. It is produced inside concrete interfaces that decide what annotators see, how options are framed, and what choices are available.

Small UI decisions can change label distributions, ambiguity, and even what kind of quality annotators notice.

So data collection is part of the alignment method, not merely a procurement step.

Chapter 11 An example of one of the earliest preference data collection interface, from Anthropic's research. Bai et al. 2022. The actual conversation is a toy conversation around what is a good example conversation for data collection. License CC-BY.

Chapter 11 Example preference data collection interface from when I was served two completions from different ChatGPT beta models. The actual completions are very close in content, showing how collecting preference data can be noisy and difficult to get exactly right.

Rankings, ratings, and multi-turn traces each lose something

Pairwise rankings are simple and robust, but they collapse nuance. Ratings preserve gradation, but are harder to calibrate. Multi-turn data is richer, but much more expensive to define well.

There is no universally best format. The choice depends on which aspects of behavior you need to capture.

This is why preference data design should start from the failure mode you are trying to fix.

Chapter 11 Example preference data collection interface from an early version of the popular LMArena benchmark.

Chapter 11 Example preference data collection interface with up or down arrow from the Allen Institute of AI's research demos.

Bias enters before modeling

Vendor contracts, labeler training, prompt sourcing, and batch iteration all affect the behavior you are measuring.

By the time a reward model sees the data, the dataset already encodes many product and organizational decisions.

In practice, good teams treat collection as an iterative pipeline, not a one-shot labeling job.

Chapter 11 Example user interface of text-to-image models.

Chapter 11 Overview of the multi-batch cycle for obtaining human preference data from a vendor. The ramp up period allows a narrowing of goals and methodology in order to create the best possible data. It is expected that a larger proportion of the data from the earlier batches will have to be thrown out due to quality issues. This is one timeline example for a smaller data contract (~$500K) and much larger data contracts can vary substantially.

Review

5 quick checks

Questions with revealed answers.

1 Why is the labeling interface so important in preference data collection?

Because it determines what annotators notice, how they compare options, and which distinctions the resulting data can capture.

2 What is one tradeoff between rankings and ratings?

Rankings are easier to collect consistently, while ratings can express more nuance but are harder to calibrate across annotators.

3 Why is multi-turn preference data harder?

Because quality depends on conversation history, role consistency, and longer-horizon judgments that are expensive and ambiguous to label.

4 Where does bias enter the pipeline before any model is trained?

In prompt sourcing, interface design, vendor setup, annotator instructions, and the iterative filtering of collected batches.

5 What is the right mental model for preference-data collection?

It is an iterative measurement system for behavior, not a simple act of gathering labels.