The Scaling Problem
Small RLHF runs fail in predictable ways that are easy to diagnose: bad rubric, mismatched annotators, noisy gold pairs. Large RLHF runs fail in insidious ways that don't show up until you're deep into training. Three failure modes dominate.
Annotator drift is the most underappreciated problem in high-volume annotation. An annotator who applies your rubric consistently at task 50 won't apply it the same way at task 2,000. Their internal calibration of what "good" looks like shifts gradually over a long engagement — through fatigue, through pattern-matching shortcuts that develop as they see more examples, through subtle rubric reinterpretation. You can't solve this with better guidelines. You solve it with recalibration checkpoints and continuous gold pair monitoring, which almost no small pilot needs and every large pipeline requires.
Inter-annotator agreement collapse is what happens when you expand your annotator pool to meet throughput demands without maintaining calibration standards. InstructGPT ran its RLHF pipeline with a carefully selected contractor pool — approximately 40 annotators for roughly 50,000 preference pairs, around 1,250 pairs per annotator over the full collection run. When Llama 2 scaled to 1M+ preference comparisons, Meta's annotation operation required fundamentally different infrastructure: tiered annotators, structured calibration, systematic IAA monitoring, and explicit quality filtering before data entered training. The teams that fail at scale try to apply the InstructGPT model to Llama 2 volumes.
Task complexity spill is subtler. When your annotation queue is small, you can hand-pick prompts suited to your rubric. At 10,000+ pairs, queue diversity expands to cover edge cases, ambiguous instructions, and multi-turn dialogues that your original rubric doesn't handle cleanly. Annotators encounter tasks that don't fit the decision tree they were trained on, make judgment calls, and introduce systematic variance the rubric wasn't designed to capture. The fix is rubric versioning — explicit updates as task types expand — not hoping annotators figure it out. For more on rubric design fundamentals, see our guide on building a preference dataset for RLHF.
Infrastructure Requirements
The infrastructure shift from 100 to 10,000+ annotations isn't just quantitative. The architecture has to change.
Annotator pool architecture at scale requires explicit tiering. A flat pool of 10–15 annotators works fine at pilot scale but becomes unmanageable at production volume. Production pipelines run 3-tier structures: gold annotators (5–10% of the pool, highest IAA, used for calibration data and disputed resolution), regular annotators (the production workforce), and spot-checkers (a separate group that reviews samples from the main pool, not part of the primary labeling flow). This structure means calibration signals are always coming from a stable reference group, not from the same pool that's drifting under volume pressure.
Batching strategy matters because quality control requires it. Annotations should be processed in batches of 200–500 pairs, with each batch moving through a QC gate before entering the training queue. Don't let 5,000 annotations accumulate and then audit them — by then you've potentially poisoned a significant portion of your training data and have no mechanism to catch the drift mid-run.
Quality control cadence at scale means auditing 10% of every batch (20–50 pairs), running calibration sessions for each annotator every 500–750 tasks, and measuring IAA continuously on a 15% redundant annotation sample. A Cohen's κ floor of 0.65 should be a hard gate — not a target to aim for, but a minimum below which batch data doesn't move to training. At 100 annotations this feels like overhead. At 100,000 it's the only thing standing between you and a reward model trained on noise.
IAA monitoring frequency is where teams get complacent. At scale, IAA should be computed per annotator, per batch, and per task type — not just once at the end of a collection run. An annotator whose κ was 0.71 in week one and has dropped to 0.58 in week four needs recalibration before they contaminate another 500 pairs, not after. For a detailed treatment of IAA metrics and when to use them, see how to build an RLHF dataset from scratch.
Annotator Sourcing at Scale
Crowd platforms fail at high-volume expert annotation in three specific ways, and each failure mode gets worse as volume increases.
MTurk and Prolific work for tasks where general English fluency and basic reasoning map to the annotation target — simple classification, sentiment labeling, basic factual judgments. They fail for RLHF preference tasks because the annotation target isn't "what does a reasonable person think" — it's "what does a qualified evaluator think, applying a specific rubric with enough domain background to apply it consistently." For general instruction-following RLHF, generalist crowds can achieve κ in the 0.55–0.65 range with strong rubrics. For domain-specific tasks (code, medical, legal, financial), that number drops to 0.35–0.45 and the labels carry systematic bias toward surface features of quality rather than actual quality. The gap doesn't close with better guidelines.
At high volume, crowd platforms add a second failure: you can't recalibrate fast enough. When you have 2,000 MTurk workers contributing annotations, running calibration sessions across that pool is operationally infeasible. With a structured expert network of 50–200 vetted annotators, calibration is a scheduled process you can run every 500–750 tasks per annotator. This is why Anthropic's annotation operations used trained contractor pools rather than open crowd platforms for reward model training data.
For scaling RLHF pipelines, a tiered sourcing structure works best. Gold annotators are your most experienced, highest-IAA annotators — typically 5–15 people from your original pilot who have demonstrated stable, consistent judgment over time. They don't do production labeling at scale; they produce gold pairs, resolve disputed annotations, and serve as calibration references. Their output is a quality control asset, not a throughput asset.
Regular annotators handle production volume. Vetting requires demonstrated domain knowledge (credential verification for specialized domains, calibration task performance for general domains), a calibration run before live annotation begins, and ongoing IAA monitoring. For RLHF on general tasks, 40–100 annotators is a manageable production pool. For specialized domains, you may need fewer but with higher credential bars. Spot-checkers form a separate reviewer cohort that never sees the primary annotation queue — their job is to re-evaluate samples from the production pool, flagging batches with systematic issues. Independent calibration is essential; a spot-checker trained alongside the production pool carries the same biases they're supposed to catch.
Quality Control Systems
At production scale, quality control can't be manual review. It has to be systematic infrastructure with defined thresholds and automatic routing.
Gold question injection is your primary continuous signal. Maintain a library of 300–500 gold pairs with pre-established labels (drawn from expert consensus or high-agreement prior annotation), and seed them into each annotator's queue at 5–8% frequency. Track per-annotator accuracy on gold pairs in a live dashboard — not aggregated, but individually. An annotator whose gold accuracy drops below 75% should be flagged for recalibration before proceeding; below 65%, their recent batch should be held from the training queue pending review. This runs continuously for the life of the annotation engagement, not just at onboarding.
IAA tracking dashboards need to surface three signals: aggregate IAA across the full pool (batch-level), per-annotator IAA trends over time (individual drift detection), and IAA by task type (rubric adequacy for different prompt categories). The task-type breakdown is the one teams most often skip — and it's where rubric gaps hide. An annotator pool with κ = 0.72 overall can have κ = 0.44 on multi-turn dialogue tasks if the rubric doesn't specify how to handle context-dependent quality judgments. For more on evaluation methodology for LLMs, see our guide to human evaluation for LLMs.
Automatic flagging of low-agreement batches means setting rules that hold batches from the training queue when they fall below quality thresholds without requiring human intervention for every flagged case. Batches where fewer than 60% of annotators agree on the majority-preference label should be automatically routed for review. Batches where average annotator gold accuracy fell below threshold during the collection window should be held. This is where the QC infrastructure pays for itself — catching bad batches before they enter training is dramatically cheaper than diagnosing reward model failures downstream.
Annotator performance scoring should composite gold accuracy (45%), IAA with the pool (35%), and task completion rate (20%) into a per-annotator quality score that updates weekly. Annotators in the bottom 15% by composite score get recalibration intervention; annotators who don't recover within two calibration cycles should be removed from the pool. Llama 2's annotation team used multi-stage annotation with human reviewers going through multiple training iterations before handling live preference data — a structured performance management approach, not a self-selected crowd.
Cost Modeling
Annotation costs at scale compound in ways that aren't obvious from small-scale pricing, and the apparent savings from crowd annotation reverse on closer inspection.
Expert preference annotation for general RLHF tasks runs $8–$18 per pair depending on task complexity, annotator tier, and turnaround. Domain-specific expert annotation (medical, legal, financial, code) runs $25–$65 per pair when credential verification and specialized expertise are required. The back-of-envelope for a 100k-pair general RLHF dataset at $12/pair is $1.2M in annotation spend. At 1M pairs — the Llama 2 scale — that's $12M+ for general tasks, substantially more for specialized domains.
Crowd annotation appears cheaper: $0.50–$2.00 per general task on MTurk, $3–$8 on Prolific. At 100k pairs, crowd annotation appears to cost $75k–$200k versus $1.2M for expert annotation. But this comparison ignores three factors that flip the math for domain-specific RLHF: reward model retraining cost when crowd-labeled data produces a miscalibrated reward signal (typically 30–50% of annotation-equivalent compute spend), dataset remediation when quality issues are caught late (often 15–25% annotation spend in re-collection), and the downstream model quality gap that requires additional RLHF cycles to close.
At 10k pairs, expert vs. crowd: roughly $120k vs. $15k. At 100k pairs: $1.2M vs. $150k. At 1M pairs: $12M vs. $1.5M. The expert premium is real — but so is the quality premium, and for domain-specific reward models, the crowd path often leads to more total spend through re-collection and retraining cycles.
The practical optimizations for scaling costs: task decomposition splits complex preference tasks into simpler sub-tasks priced at lower tiers, concentrating expert judgment on the final preference call rather than the full evaluation. Tiered review routes high-confidence pairs (strong annotator agreement) through the standard pipeline and escalates disputed pairs to gold annotators — reducing blended cost per pair by 25–35%. Hybrid human/AI pre-filtering uses an existing reward model or lightweight classifier to identify hard cases requiring careful human judgment, routing easy cases (large, unambiguous quality gaps between responses) to lower-cost annotation tiers. At 10k pairs these optimizations add overhead. At 100k+, they're the difference between a sustainable annotation budget and one that gets cut mid-project.
Platform vs. In-House
The build-vs-buy decision for large-scale RLHF annotation comes down to four criteria, and the thresholds are higher than most teams expect.
Turnaround SLA requirements favor in-house when you need annotation results within 24–48 hours at volume. Marketplace platforms typically operate on 5–15 business day SLAs for large batches. If your training pipeline is continuous — ingesting preference data from production traffic and retraining on a rolling basis — that latency may be acceptable. If you're running scheduled RLHF cycles with tight training windows, in-house gives you scheduling control.
Domain coverage is where marketplace platforms have a structural advantage over in-house builds. Assembling a credentialed, calibrated annotator pool for a specialized domain takes 3–6 months of recruiting, vetting, and training. A marketplace with an existing expert network in your target domain collapses that timeline to weeks. This matters most for teams running RLHF across multiple specialized domains — the operational cost of maintaining separate expert pools for medical, legal, financial, and code annotation in-house is substantial.
Compliance requirements can force the in-house decision. HIPAA-compliant annotation operations require Business Associate Agreements, data handling protocols, and audit capabilities that most annotation marketplaces support but require careful vetting. For annotation involving sensitive business data, proprietary IP, or regulated content, in-house control over the data pipeline may be non-negotiable regardless of cost.
Cost per annotation at target scale is where the arithmetic often surprises teams. In-house annotation operations have fixed costs — recruiting, training, tooling, QC infrastructure, management overhead — that make them uneconomical below ~50k pairs/year. Above that volume, in-house cost per pair often falls below marketplace pricing. The crossover point varies by domain and annotator tier, but the general rule: at 10k pairs, buy. At 100k+ pairs/year with stable domain requirements, the build case strengthens significantly.
Most teams running RLHF at 10k–50k pairs scale should not be building their own annotation operations. The fixed cost amortization doesn't work, the operational complexity is underestimated, and the time to a calibrated, quality-producing pool is longer than teams expect. Using a marketplace at this scale — while focusing internal resources on rubric design, quality criteria, and training pipeline — produces better annotation quality faster and at lower total cost.
Start with Benchmarking Before You Scale
Before committing to a large-scale RLHF build — whether in-house or through a platform — the highest-leverage step is benchmarking your annotation pipeline against high-quality expert data. Running your reward model training on a verified expert preference dataset tells you what your current annotation quality is actually producing, what IAA your rubric is realistically achieving, and what your reward model accuracy looks like with expert-labeled versus standard-labeled training data. That benchmark determines whether your annotation spend at scale produces a better model or just a larger dataset with the same quality ceiling.
This is where most teams skip a step: they assume their small-pilot annotation quality represents what they'll get at scale, and plan their budget accordingly. It usually doesn't. Running a quality benchmark before committing to a 100k-pair build is the cheapest way to discover that your rubric needs work, your annotator pool isn't achieving the IAA you expected, or your task decomposition is creating noise that a larger dataset will amplify, not reduce.