RLHF — Reinforcement Learning from Human Feedback — is the training stage that transformed base language models into GPT-4, Claude, and Gemini. It's also the stage most teams underestimate. Supervised fine-tuning is hard but familiar: write prompts, write completions, run the training loop. Preference data collection is different in kind. You're encoding human judgment, and when that judgment is poor — vague criteria, mismatched annotators, inconsistent standards — the reward model learns confidently in the wrong direction. A sycophantic model that agrees with everything. A model that prefers longer answers over accurate ones. A model that sounds authoritative while being wrong.
This post is a practical, step-by-step guide to building a preference dataset that produces useful reward signal. We'll cover evaluation criteria design, annotator sourcing, interface design, quality control, and dataset sizing — with real numbers from published RLHF work where they exist.
What Is a Preference Dataset?
A preference dataset is a collection of pairwise comparisons: for a given prompt, a human annotator reviews two model outputs and indicates which is better. The output is a label — chosen and rejected in the canonical format used by TRL and HuggingFace — that the reward model trains on.
This is structurally different from supervised fine-tuning data. In SFT, you give the model examples of the right answer. In preference collection, you give it a signal about which of two answers is closer to right. The model doesn't see a ground-truth response — it learns a scalar value function from human judgments. That distinction has enormous downstream consequences: the reward model is only as good as the quality and consistency of those judgments. Noisy preference labels don't just add variance; they actively teach the reward model to optimize for the wrong thing.
OpenAI's InstructGPT paper made this explicit. Their annotator team wasn't a general crowd pool — they were contractors selected for English fluency, sensitivity to potentially harmful content, and the ability to articulate reasoning behind preferences. Anthropic's Constitutional AI work added another layer: principles-guided preference collection, multiple review passes, and deliberate red-teaming. Neither team was optimizing for annotation throughput. They were optimizing for signal quality.
Step 1 — Define Your Evaluation Criteria
The most common failure in RLHF data collection happens before a single annotation is collected: the criteria are too vague.
"Helpfulness, harmlessness, and honesty" — the 3H framework popularized by Anthropic — is the right starting point, but it's not sufficient by itself. "Helpful" means different things for a customer service assistant versus a coding assistant. An annotator evaluating a coding response needs to distinguish between code that compiles and code that's correct, idiomatic, and maintainable. An annotator evaluating a clinical response needs to distinguish between an answer that sounds medically plausible and one that's actually accurate.
Customer service assistant rubric
Helpfulness = directly addresses the customer's stated issue; doesn't require a follow-up question to understand the answer; uses plain, non-jargon language; takes appropriate escalation action when needed. A technically accurate response that escalates the customer's frustration loses to one that resolves the situation — even if the latter is less technically precise.
Coding assistant rubric
Helpfulness = code runs without modification on the stated environment; follows the language's idiomatic style; handles the edge case described in the prompt; explanation matches the user's apparent experience level. A response with elegant architecture that silently fails on the edge case loses to a simpler one that handles it correctly.
Write your rubric as a decision tree, not a set of vague principles. Annotators need to know — when Response A is more thorough but Response B is more accurate, which attribute wins. If you don't define this, you'll get inconsistent decisions and low inter-annotator agreement, and a reward model trained on noise.
Step 2 — Source the Right Annotators
This is where RLHF projects succeed or fail. Annotator selection isn't just a quality problem — it's a validity problem. Preference data collected from mismatched annotators doesn't have more noise; it has systematic bias that your reward model will faithfully learn.
General crowdsourcing platforms — MTurk, Prolific — work fine for perceptual tasks, simple classification, and domains where "a reasonable person" judgment is the actual target. They fail badly for technical and sensitive domains. A crowd worker evaluating a medical response cannot tell the difference between a plausible-sounding answer and a clinically accurate one. They will, reliably, prefer responses that sound authoritative over responses that are correct. Your reward model learns to optimize for sounding authoritative.
The annotation quality gap is not marginal. Research on legal annotation quality found that Cohen's κ dropped from 0.74 with bar-admitted attorneys to 0.41 with non-specialist annotators on the same task. The non-specialists weren't careless — they simply lacked the background to make consistent, well-founded judgments.
The cost difference between expert annotators and crowd workers is real, but the arithmetic works out. Twenty expert annotations producing trainable signal beat 200 crowd annotations producing reward model noise.
Step 3 — Design the Annotation Interface
The annotation interface shapes data quality in ways that are easy to underestimate. Three design decisions matter most.
Eliminate positional bias
Annotators develop a systematic preference for the first response shown — the primacy effect — or for whichever response appears on the left in a side-by-side layout. This isn't conscious; it's a cognitive artifact of visual attention. The fix is simple: randomly swap which response appears in position A and which in position B, independently for each task. Record which response was in which position so you can detect and correct for residual positional effects in your final dataset.
Pairwise comparison vs. absolute scoring
For most RLHF pipelines, pairwise A/B comparison outperforms absolute scoring. The reason is calibration: annotators are much better at relative judgments ("this is better than that") than at absolute ones ("this is a 4 out of 5"). Absolute scores require annotators to maintain a stable internal scale across hundreds of tasks, which they don't do consistently. For best-of-N ranking tasks where you need a total ordering, combining absolute scoring with pairwise comparison on adjacent pairs is more robust.
Inter-annotator agreement targets
For preference tasks, target Cohen's κ ≥ 0.6 before scaling up collection. A κ below 0.5 indicates that your rubric is ambiguous, your annotators lack the background for the task, or your response pairs are too similar to distinguish reliably. κ in the 0.65–0.80 range is achievable with well-designed tasks and qualified raters. Above 0.80 on preference data usually means your pairs are too easy — you're not collecting useful discrimination signal. Collect redundant annotations on 10–15% of your dataset and measure κ continuously.
Step 4 — Quality Control at Scale
Scaling annotation volume without scaling quality control produces a dataset that looks large and trains poorly. Build these mechanisms in from day one.
Gold standard pairs
Before live annotation, construct 100–200 pairs with pre-established correct answers — drawn from expert consensus or high-agreement prior labels. Seed these into your annotation queue at ~5–10% frequency. Track each annotator's accuracy on gold pairs separately from live work. Consistent accuracy below 75% on gold pairs means the annotator is gaming the system or lacks the background for the task. Run this continuously, not just at onboarding.
Annotator calibration sessions
Before any annotator labels live data, run a calibration session: they annotate 30–50 pre-labeled pairs, receive feedback on disagreements, and discuss the reasoning behind the rubric. This dramatically reduces early annotation variance and surfaces rubric ambiguities before they contaminate your full dataset.
Handling disagreement
When multiple annotators label the same pair and disagree, majority vote works for tasks where the goal is "what do most qualified people think." Weighted expert opinion — giving more weight to annotators with consistently higher IAA — is better when your pool has quality variance. For genuinely ambiguous pairs where agreement is low even among experts, the right call is often to exclude the pair from training entirely rather than forcing a noisy label.
Red flags to monitor
Response length bias (annotators systematically prefer longer responses — check correlation between length difference and preference label). Annotator fatigue (completion time decreasing significantly over a session — set session length limits). Position persistence (an annotator choosing "A" more than 60% of the time regardless of content).
Step 5 — Dataset Size and Diversity
How much preference data do you actually need? Published RLHF papers give useful reference points. InstructGPT trained its initial reward model on approximately 50,000 preference pairs covering diverse instruction-following tasks. Llama 2's RLHF stage used roughly 1 million binary comparisons for its final reward model — but Llama 2 is a general-purpose model at massive scale. For domain-specific reward models, effective training has been demonstrated with as few as 5,000–10,000 high-quality pairs when the task scope is narrow and annotator quality is high.
The more important constraint is diversity, not volume. A reward model trained on 20,000 pairs from a narrow prompt distribution generalizes poorly to out-of-distribution requests. Coverage gaps are where reward model failures cluster in production. Before scaling annotation volume, audit your prompt set for:
- Distribution of task types (instruction following, factual Q&A, creative, code, dialogue)
- Difficulty distribution — include genuinely hard pairs where responses are close in quality
- Edge cases and adversarial inputs designed to trigger failure modes
- Domain coverage relative to your target deployment distribution
The practical stopping criterion: measure reward model performance on a held-out validation set after each 1,000–2,000 pairs. When validation accuracy plateaus over three consecutive checkpoints, you're in diminishing returns. More data at that point is less valuable than auditing your training distribution for coverage gaps.
Common Mistakes That Sink RLHF Projects
Most RLHF failures aren't mysterious. They come from the same recurring mistakes.
Annotation instructions that are too vague
"Rate which response is more helpful" without a rubric produces annotators falling back on personal heuristics — response length, surface confidence, formatting. Your reward model learns those heuristics. Write decision trees, not principles.
A homogeneous annotator pool
Preference data from a single demographic, cultural, or professional background encodes a single worldview into your reward signal. This produces a model that's well-aligned for that worldview and systematically miscalibrated for everyone outside it.
Not filtering for annotator consistency
Collecting labels without tracking per-annotator IAA means low-quality annotators contaminate your full dataset. Label filtering after the fact is expensive. Build consistency monitoring in from the start.
Treating preference data collection as a one-time step
Your model's behavior changes through fine-tuning and deployment. Edge cases that weren't in your original distribution will emerge in production. The teams running the best models continuously collect preference data across production traffic — the pipeline stays active, not shut down after the initial training run.
Putting It Together
High-quality preference data collection is a multi-stage engineering problem: rubric design, annotator sourcing, interface design, quality control, and coverage management. Each step has failure modes that compound downstream. A reward model is only as good as the human judgment that trained it, and building that judgment pipeline correctly is the difference between a model that improves with RLHF and one that confidently gets worse.
Start with criteria before annotators. Start with annotators before interface. Build quality control before you scale. Audit coverage before you scale further. And treat the pipeline as ongoing infrastructure, not a project to check off.