Why RLHF Data Is Different
Supervised labels are largely deterministic: is this entity a person or an organization? With the right guidelines and enough examples, a well-run annotation process converges. RLHF data is fundamentally different — it encodes relative human preferences, and those preferences are context-dependent, domain-dependent, and rater-dependent in ways that break standard annotation pipelines.
OpenAI's InstructGPT paper was explicit about this. The annotation team wasn't sourced from a general crowd pool. They were contractors selected for English fluency, sensitivity to harmful content, and the ability to provide nuanced explanations for their preferences. Anthropic's Constitutional AI work built additional structure on top of this — principles-guided preference collection, red-teaming, and multiple review passes. Neither team was optimizing for throughput. They were optimizing for signal quality.
A good preference judgment requires: understanding what the prompt was actually asking, reading both responses carefully to distinguish subtle quality differences, applying a consistent standard of what "better" means, and recognizing failure modes that are easy to miss (factual errors, safety violations, logical inconsistencies that sound plausible). A crowd worker completing 150 tasks per hour cannot do this. Standard annotation pipelines built for throughput produce preference data that trains reward models to prefer longer answers, more confident-sounding answers — whatever pattern raters latched onto fastest. This is how you get sycophantic fine-tunes.
Step 1: Define Your Task Type
Before you write a single prompt or recruit a single annotator, get clarity on what kind of preference data you're collecting. The task type determines everything downstream.
Instruction following
The most common starting point. Annotators judge which response better satisfies a user request. Requires raters who can evaluate factual accuracy and helpfulness across diverse domains. Generalist raters work here if your domain is truly general.
Dialogue ranking
Multi-turn conversations evaluated holistically. Raters need to track context across turns and evaluate coherence alongside response quality. More cognitively demanding than single-turn preference.
Safety evaluation
Raters identify harmful, toxic, or policy-violating content. Requires emotional resilience and clear understanding of your safety policies. Do not treat this as a general annotation task.
Code review
Ranking code responses for correctness, efficiency, and style. Requires annotators with real programming ability — not 'familiarity with code.' A rater who cannot mentally trace execution cannot give useful signal on a coding task.
Domain-specific preference
Medical, legal, scientific, or financial content. This is where annotator mismatch does the most damage. If your model is being trained to assist oncologists, you need oncologists in the loop.
Step 2: Design Your Preference Pairs
Prompt construction is where most teams underinvest. The quality of your preference pairs is upstream of everything else.
Prompt construction
Ensure your prompt set is diverse across task types, difficulty levels, and edge cases. Over-indexing on easy, common prompts produces a reward model that's well-calibrated on easy cases and useless on hard ones. Deliberately seed your prompt set with ambiguous instructions, multi-step requests, and adversarial inputs.
Response sampling strategy
Don't use the same model for both responses in a pair. Generate response A from one checkpoint and response B from a different checkpoint — or vary temperature and sampling parameters. If both responses are too similar, raters default to arbitrary choices. If both are obviously one good/one bad, you're collecting easy labels that don't train a useful reward model. You want distinguishable responses where preference requires real evaluation.
A/B vs. best-of-N
For most RLHF pipelines, pairwise A/B comparison is the right format. Best-of-N (ranking 3–5 responses) can work for reward model training but introduces combinatorial annotation burden and makes tie-breaking harder. If you go best-of-N, define your tie-breaking convention explicitly and enforce it in the interface.
Tie-breaking
Allow ties but require a written rationale, then filter tied pairs out of the training set. This avoids forcing noise-amplifying decisions on genuinely ambiguous pairs while still capturing where your model's uncertainty should live.
Step 3: Select the Right Annotators
This is where RLHF data collection either works or doesn't. The annotator selection problem in RLHF is not just a quality problem — it's a validity problem. Preference data collected from the wrong raters doesn't just have more noise; it has a systematic bias that will be faithfully learned by your reward model.
Domain expertise beats volume. A hundred crowd workers producing medical preference data are strictly worse than ten domain-matched clinicians. The crowd workers cannot catch factual medical errors. They will prefer responses that sound authoritative over responses that are clinically accurate. Your reward model learns to optimize for sounding authoritative.
Annotator agreement thresholds: For preference tasks, target an inter-annotator agreement (Cohen's κ) of ≥ 0.65 before scaling up. Anything below 0.60 is a sign that either your task design is unclear or your annotators don't have the background to make consistent judgments. Kappa in the 0.65–0.80 range is achievable on well-designed preference tasks with qualified raters. Above 0.80 on preference data is rare and often indicates your response pairs are too easy.
For domain-specific tasks — medical, legal, scientific, financial — require verified domain background, not self-reported expertise. The signal you're collecting is only as good as the judgment behind it.
Step 4: Build Your Annotation Interface
The interface design affects data quality more than most teams realize. Three principles:
Show both responses without anchoring bias
Randomly swap which response appears first (A/B order) and don't display any model metadata alongside the responses. Position bias is real — left/top preference shows up in large annotation studies. Randomize and account for it.
Capture rationale, not just votes
A preference vote with no explanation is weak signal. Requiring raters to write even a one-sentence explanation dramatically improves label quality — it forces a reasoning step before selection and gives you audit material when labels look suspect. This is slower and more expensive. It is worth it.
Show the prompt in full context
For multi-turn tasks, display the full conversation history. For tasks with system prompts, show the system prompt. Raters who only see a response without context cannot evaluate instruction-following. This sounds obvious and is frequently violated in practice.
Step 5: Quality Control at Scale
Collecting preference data at volume creates systematic failure modes. Build quality control in from the start.
Golden sets
Seed your annotation queue with items that have pre-established correct answers (from expert consensus or very high-agreement prior labels). Track rater accuracy on golden set items. A rater consistently below threshold gets flagged or removed. Run this continuously, not just at onboarding.
Red-team your pipeline
Before full-scale collection, run a deliberate adversarial pass. Can raters cheat by always picking the shorter answer? The first answer? The answer with more bullet points? If these heuristics produce above-chance accuracy on your golden set, your task design has a flaw that will be exploited.
Handling adversarial raters
Speed-based detection (completion time below threshold triggers review) and pattern-based detection (consistent position preference, always-tie patterns) catch most adversarial raters. Remove them early — their labels have negative value.
Dataset audits
Before training, check: label distribution by rater, agreement rate by rater, completion time distribution, and rationale quality. A rater with 30% disagreement rate relative to the pool average is worth examining. A rater with average completion time of 8 seconds per pair did not read the responses.
Step 6: Format for Training
The de facto standard for RLHF preference pairs is the chosen/rejected format, used by both the TRL library and the HuggingFace datasets ecosystem.
{
"prompt": "Explain L1 vs L2 regularization.",
"chosen": [
{"role": "user", "content": "Explain L1 vs L2 regularization."},
{"role": "assistant", "content": "L1 adds absolute weight values to the loss, producing sparse solutions where some weights go to zero. L2 adds squared weights, penalizing large weights but rarely zeroing them out. Use L1 when many features are irrelevant; L2 as the default for most neural network training."}
],
"rejected": [
{"role": "user", "content": "Explain L1 vs L2 regularization."},
{"role": "assistant", "content": "Both L1 and L2 are regularization techniques that help prevent overfitting by adding a penalty term to the loss function."}
]
}TRL's DPOTrainer and RewardTrainer both consume this format natively. For multi-turn data, include the full conversation in both chosen and rejected, not just the final assistant turn. The reward model needs context.
Store metadata alongside each pair: annotator IDs (anonymized), agreement rate, completion timestamp, and any collected rationale. This metadata is not used in training, but is essential for dataset audits and for filtering by quality tier when you have mixed-quality annotations.