Why Annotator Prompts Determine Your Reward Model Ceiling
The annotation prompt is the contract between your training signal and the human's judgment. Every ambiguity in that contract produces variance in the output — and variance at the annotation layer compounds through reward model training into a misaligned policy. This is the failure mode that doesn't announce itself until you're in RLHF iteration 3 wondering why your model got worse.
The concrete version: we've seen teams run the same 200 response pairs through two annotation setups. The first used a single-sentence instruction: "Choose the better response." The second used a structured rubric with weighted dimensions, decision rules for ties, and two anchor examples. The first returned Fleiss' κ = 0.38 — agreement barely above chance on preference tasks. The second returned κ = 0.72. Same annotators. Same pairs. The prompt was the variable.
Low IAA (inter-annotator agreement) doesn't just mean noisier reward model training — it means your reward model is learning a mixture of different annotators' private heuristics, none of which you specified. Annotator A weights brevity. Annotator B weights technical completeness. Annotator C penalizes hedging language. Your reward model learns some linear combination of those idiosyncratic preferences and you have no way to know which one it's optimizing after the fact. See also: how to build a preference dataset for RLHF for the broader pipeline context this fits into.
The Three Most Common Annotator Prompt Failures
1. Ambiguous criteria — "better" without defined dimensions
Before
"Review the two responses below and choose which one is better. If they are equal, mark 'Tie'."
After
"Evaluate both responses on four dimensions: factual accuracy (40%), completeness — does it address all parts of the question (30%), appropriate tone for the user's apparent expertise level (20%), and safety — no harmful or misleading content (10%). Choose the response with the higher weighted score. If scores are within 5 points of each other, mark 'Tie'."
The before version produces annotators who fall back on whatever they've been implicitly trained by experience to call "better." Usually that means length, surface confidence, or formatting — none of which you specified as signal.
2. Missing edge case handling
Before
"If both responses are equally good or equally bad, mark 'Tie'."
After
"Edge cases: (1) Both responses are equally good → Tie. (2) Both responses contain a factual error → mark the one with fewer errors as preferred; if errors are equivalent, Tie. (3) Response A is more accurate but Response B has better tone → use the weighted scoring formula above; accuracy outweighs tone. (4) One response is off-topic → mark the on-topic response as preferred regardless of quality. (5) You're genuinely unable to distinguish after 90 seconds → mark 'Cannot Determine' with a note."
3. Inconsistent grounding — no anchor examples
Before
"A helpful response is one that directly addresses the user's question, is factually accurate, and uses appropriate language."
After
Same definition — followed by two concrete examples: one where Response A clearly wins (A answers the question with a correct code snippet; B gives a correct but vague explanation), one where Response B clearly wins (B is factually accurate; A confidently states wrong information), and one ambiguous case where both have partial credit, with the expected answer and the reasoning for it.
Without anchor examples, every annotator independently calibrates what "appropriate language" and "directly addresses" means. You get five annotators with five internal scales that diverge as the task set gets harder.
The Anatomy of a High-Quality Annotation Prompt
A production-grade annotation prompt has five components. Skipping any one of them is where κ drops. For the full dataset pipeline context, see how to build an RLHF dataset from scratch.
(a) Task framing sentence
One sentence describing what the model is supposed to do in this context — not what the annotator is supposed to do. Example: "The model is acting as a customer support assistant for a SaaS product. It should help users resolve issues, escalate appropriately, and never promise features that don't exist." This frames the annotator's judgment within the model's intended use case, not an abstract preference.
(b) Evaluation dimensions with weights
Named dimensions with explicit percentage weights. A working default for general instruction-following: accuracy 40%, completeness 30%, tone 20%, safety 10%. Adjust for domain — a coding task might shift to correctness 50%, completeness 30%, style 20%, with safety as a disqualifier rather than a weighted dimension. The weights force annotators to make the tradeoff you intended, not the one that felt natural to them.
(c) Decision rules for ties and edge cases
At minimum: what to do when both responses contain errors, when one dimension wins on each side, and when the annotator genuinely cannot distinguish. Enumerate these explicitly rather than relying on annotators to extrapolate from the rubric. A decision tree is better than prose — annotators are making fast decisions and need to find the rule quickly.
(d) Anchor examples (minimum 2, ideally 3)
One clear A-wins example with reasoning. One clear B-wins example with reasoning. One genuinely ambiguous example with the expected label and an explanation of how to resolve the ambiguity using the rubric. Anchors do more work than any other component — they translate the rubric from abstract principles into the calibrated judgment the annotator needs to replicate.
(e) Annotation output format — label + required rationale
The output format should require a preference label (A / B / Tie / Cannot Determine) and a free-text rationale of 1–3 sentences. Rationale is not optional. It surfaces disagreement patterns early — when you see annotator rationales diverging on the same pair, that's a rubric gap, not an annotator quality issue. It also provides interpretability signal for reward model debugging and, in some pipelines, training signal for the reward model itself.
Calibration Rounds: How to Test Your Prompt Before Full Rollout
No annotation prompt should go to a full cohort without a calibration round. Run a batch of 50–75 pairs with 3+ annotators, compute Fleiss' κ across all annotators on all pairs, and apply the following thresholds before scaling:
When κ < 0.60, run a triage checklist before rewriting. The failure is almost always in one of three places:
- Criteria definition failure: annotators disagree on what the dimensions mean. Fix: add more precise language and a negative example (what is explicitly not a factor).
- Anchor failure: anchor examples aren't close enough to the hard cases in the live task set. Fix: build new anchors from the highest-disagreement pairs in your calibration batch.
- Edge case coverage failure: the pairs include situations the rubric doesn't address. Fix: enumerate new edge case rules from the disagreement patterns in annotator rationales.
The calibration round is not a nice-to-have. It's insurance against wasting 10,000 annotations on a broken task definition. In our experience, teams that skip calibration and scale directly spend 2–3× more time fixing downstream reward model behavior than teams that ran a 50-pair calibration batch first. For scale considerations on larger runs, see scaling RLHF to 10,000+ annotations.
Domain-Specific Considerations
The five-component structure above applies universally. What changes by domain is which dimensions to weight, what edge case rules to add, and how to structure the output format.
Coding tasks
Add a correctness-first hierarchy: annotators should evaluate whether the code is functionally correct before considering style, completeness, or explanation quality. A response with elegant architecture that fails on the stated edge case loses to a clunky one that handles it correctly. Make this explicit: "If Response A contains a functional error and Response B does not, prefer B regardless of other dimensions." Correctness is a disqualifier, not a weighted factor.
Medical and legal content
Add an explicit "defer to ground truth" rule. The annotation task is preference — which response is more helpful, accurate, and appropriately scoped — not clinical or legal interpretation. Annotators should not be making judgment calls about whether advice is medically sound based on personal knowledge; they should be evaluating whether the response correctly defers to professional judgment, avoids overconfident claims, and stays within appropriate scope. The rubric should also include a safety disqualifier: any response that gives specific medical or legal advice without appropriate disclaimers loses automatically, regardless of accuracy.
Conversational and chat tasks
Add turn-level coherence as an explicit dimension. In multi-turn conversations, a response that's technically accurate but ignores the thread of the conversation — failing to acknowledge what the user said two turns ago, not maintaining established context — is worse than a slightly less precise response that tracks the conversation correctly. Without this dimension, annotators evaluate each response pair in isolation and miss the coherence signal entirely.
Long-form content
Break the rubric into segments rather than having annotators evaluate holistically. For a 1,000-word article response, holistic evaluation produces high variance — annotators anchor on different parts of the text. A rubric with separate scores for argument structure, factual accuracy, evidence quality, and writing clarity produces much higher IAA because annotators are evaluating bounded, comparable dimensions rather than forming a single global impression. This adds annotation time per pair, but the κ improvement makes it worth it.
The Iterative Refinement Loop
A single calibration round is the minimum. Production-grade annotation pipelines run three rounds of prompt refinement with calibration checkpoints between each. The loop:
- Run calibration batch (50–75 pairs, 3+ annotators). Compute κ. If κ ≥ 0.70, proceed to full cohort. Otherwise:
- Review the highest-disagreement pairs with annotators directly — not to adjudicate who was right, but to surface what each annotator was optimizing for when they disagreed. The patterns are almost always instructive: annotator A consistently preferred brevity; annotator B was lenient on factual errors in cases where tone was strong; annotator C applied the rubric literally rather than in context. Each of these reveals a missing criterion.
- Revise the prompt to address the gaps. Usually this means adding a new edge case rule, refining the weight on one dimension, or adding an anchor example drawn from the disagreement cases.
- Re-run calibration on a fresh 50-pair batch (not the same pairs — you want new test cases, not annotators memorizing the previous batch). Measure κ improvement. Repeat if needed, up to three rounds.
Save every prompt version with its κ score. Version history isn't just housekeeping — it's how you prove data quality to a skeptical model lead. "We ran three calibration iterations; the final prompt reached κ = 0.71 on 75 calibration pairs before we scaled" is a specific, defensible claim. "We had good annotators" is not. The κ progression also tells you where your prompt refinements had diminishing returns — if round 3 moved κ from 0.68 to 0.69, you're in annotator variance territory, not prompt design territory. For the broader context on why human evaluation quality matters here, see LLM evaluation benchmarks vs. human evaluation.
Prompt Library and Getting Started
The prompt design work described here — rubric construction, anchor example selection, edge case enumeration, calibration round design — takes real time. For a first annotation task, budget 3–5 days of focused prompt engineering before you start collecting data. For domain-specific tasks (medical, legal, financial), add credential verification and domain expert review of the rubric itself.
Most teams underestimate this work because they've conflated "writing the annotation task" with "writing the model's system prompt." They're different problems. The model's system prompt shapes what the model outputs. The annotation prompt shapes what signal you extract from human judgment of those outputs. The annotation prompt is upstream of your reward model — which is upstream of your policy. Getting it wrong has a long tail.
A few reference points from published work: InstructGPT's labeler training took multiple sessions before live annotation; Anthropic's Constitutional AI work involved explicit rubric development and multi-round calibration before scaling the preference collection pipeline; Llama 2's preference data quality guidelines ran to multiple pages with explicit tie-breaking rules and ranked-preference instructions, not a single sentence. These are the teams whose RLHF results held up. The correlation between prompt design rigor and reward model quality is not accidental.