1. Why Annotation Briefs Fail
The average annotation project that goes wrong doesn't fail because the vendor delivered bad work. It fails because the brief didn't tell the vendor what good work looks like. The result is a 15–30% rework rate, a 2–3x cost overrun, and weeks of delay while the team re-annotates samples and revises rubrics under schedule pressure. The vendor didn't fail. The brief did.
Three failure modes account for most of the rework:
Ambiguous quality criteria
The brief says "annotate helpful responses" but provides no rubric. One annotator marks a response helpful if it answers the question. Another marks it helpful if it answers the question, cites sources, and avoids hedging. A third treats verbosity as a proxy for helpfulness. None of them are wrong given the brief. The resulting dataset reflects three different definitions of the target behavior — and the reward model trained on it learns a blended signal that doesn't match what you wanted.
Missing edge cases
The brief is tested on clean, clear examples from the training distribution. The vendor calibrates on them, achieves solid IAA on the calibration set, and proceeds with confidence. Then the annotations arrive and 20% of labels on production examples — the ones with ambiguous prompts, refusal responses, near-identical output pairs, or domain-specific knowledge requirements — are inconsistent or wrong. The edge cases were always in the data. The brief just didn't acknowledge them.
No IAA targets
The brief specifies a task and a rubric but no inter-annotator agreement threshold. The vendor delivers 10,000 preference pairs. You run IAA analysis and find κ = 0.45 on helpfulness ratings — far below the κ ≥ 0.70 your reward model training requires. The vendor followed the brief. The brief just didn't tell them what quality looks like at the dataset level. Now you need to re-annotate 30–40% of the batch.
A complete annotation brief eliminates all three failure modes before the first annotation task is assigned. It is the document that pays for itself — typically in the first batch, before a single rework cycle begins. For background on what annotation quality costs when briefs fail, see our AI annotation pricing guide.
2. What a Complete Annotation Brief Contains
Eight fields. All required. If any one of them is missing or underspecified, the vendor will interpret it — and their interpretation will be different from yours.
Task definition
One sentence: what you want annotators to do — not what you want to train. "Rank three AI assistant responses to the same user question by overall quality" is a task definition. "Help us improve our RLHF reward model" is a training objective. The annotator needs the former, not the latter. Keep it to one sentence. If it requires two sentences, the task is underspecified or actually two tasks.
Input format
What the annotator sees, exactly. Describe every component: the user prompt, the model completion(s), any context document, any metadata. Specify format: plain text, HTML, markdown, JSON. Include at least two concrete examples of real inputs — preferably drawn from your actual data distribution, not sanitized illustrative examples. Annotators calibrate on what they see, so what they see in the brief should match what they see in the task.
Output format
The exact label options, rating scale, or preference selection the annotator produces. No "etc." If the output is a 5-point scale, define all 5 points. If it's a preference selection, define all valid options including ties and abstentions. If it's a free-text rationale, specify minimum length and required components. Ambiguity in the output format is what produces the most dangerous class of annotation errors: invalid or structurally inconsistent labels that require custom parsing before the data is usable.
Quality criteria (rubric)
The rubric that defines what good annotation looks like, with behavioral anchor examples for each score point. Not "5 = excellent, 3 = mediocre, 1 = poor" — that is not a rubric, it is adjective substitution. A rubric provides observable behavior at each score point: what does a response that earns a 5 actually do? What specific behaviors distinguish a 3 from a 4? Include at least two anchor examples per score level, ideally drawn from real data.
Edge case handling
At least five known hard cases, each with the specified label and the reasoning behind it. Edge cases are not exceptions to the task — they are the hardest version of the task. Refusal responses. Near-identical quality pairs. Factually incorrect responses that are stylistically polished. Ambiguous prompts where the correct interpretation isn't clear. Your vendor will encounter all of these. If the brief doesn't address them, annotators will improvise — inconsistently.
IAA target
Inter-annotator agreement threshold by dimension, specified as Cohen's κ (or Fleiss' κ for 3+ annotators). Example: κ ≥ 0.70 for helpfulness ratings, κ ≥ 0.85 for safety flags, κ ≥ 0.65 for overall preference selection. Include the adjudication protocol: when annotators disagree, how is the final label determined? Senior annotator review? Majority vote? Expert tiebreak? Vendors who know their IAA target can calibrate their annotator pool and flag batches that fall below threshold before delivery.
Annotator requirements
Domain expertise, language proficiency, credentials, and any other qualifications that affect annotation quality for this specific task. "College-educated English native speakers" is a minimum floor. For domain-specific tasks — medical, legal, financial, software engineering — specify the credential or background: "MD or equivalent clinical training," "5+ years software engineering experience," "licensed attorney." Annotator requirements are not bureaucratic — they are the quality mechanism for tasks where domain knowledge determines label accuracy.
Volume + timeline
Total annotation count, daily velocity target, and milestone checkpoints. Not "10,000 labels by end of month" — that is a delivery date, not a timeline. A usable timeline specifies: calibration batch size and deadline, production batch checkpoints (e.g., 20%, 50%, 80% review points), final delivery date, and what happens if a batch fails IAA review. Volume without velocity means the vendor can front-load or back-load the work in ways that affect quality monitoring.
3. RLHF Preference Annotation Brief: Worked Example
Here is a complete filled-in annotation brief for an RLHF preference annotation task. This is the format you would send to a vendor. Every field is specified; nothing is left to interpretation. For guidance on how to structure the RLHF dataset this feeds into, see our guide to building an RLHF dataset from scratch.
Annotation Brief: RLHF Preference Annotation
Complete example — ready to send to a vendor
Task definition
Compare two AI assistant responses to the same user question. Select the response you prefer and provide a rationale.
Input format
Each annotation task presents: [User prompt] — the original user question, unmodified. [Response A] — first model completion, plain text. [Response B] — second model completion, plain text. Responses are displayed in randomized order. Example input: User prompt: "What's the difference between L1 and L2 regularization in machine learning?" Response A: "L1 regularization adds the absolute value of coefficients to the loss function, which can drive some coefficients to exactly zero, producing sparse models. L2 adds the squared values, which shrinks all coefficients but rarely zeroes them out. L1 is useful for feature selection; L2 is better for preventing overfitting when you want all features to contribute." Response B: "Regularization prevents overfitting. L1 uses the absolute value of weights and L2 uses the square. They're both used in machine learning to reduce model complexity."
Output format
Annotators produce four outputs per task: 1. Preference selection: A | B | Tie (select exactly one; Tie only when responses are genuinely indistinguishable in quality — not as a default) 2. Helpfulness rating: 1–5 for both Response A and Response B (see rubric below) 3. Accuracy flag: Y/N for both responses (Y = factual error present; N = no detected error) 4. Safety flag: Y/N for both responses (Y = safety concern present; N = none) 5. Rationale: minimum 2 sentences explaining the preference selection, referencing specific behaviors in the responses
Quality criteria — Helpfulness rubric
5 — Directly answers the question with accurate information, appropriate depth, and clear structure. Anticipates likely follow-up needs. Could be published as-is in a high-quality knowledge base. 4 — Directly answers the question with accurate information. Minor gaps in depth or structure that don't materially affect usefulness. 3 — Partially answers the question. Either accurate but incomplete, or complete but with minor inaccuracies. The user would need to look elsewhere to fully resolve their question. 2 — Addresses the topic but does not answer the question. May be tangentially relevant or accurate but non-responsive. 1 — Does not address the question. Actively misleading, entirely off-topic, or empty of information.
Edge case handling
Refusal responses: If one response is a refusal ("I can't help with that") and the other is a helpful answer to a benign question, prefer the helpful answer. Label the refusal A=1 on helpfulness unless the refusal is itself helpful (e.g., the question is genuinely harmful). Near-identical quality: Use the Tie option only when responses are substantively equivalent — same accuracy, same depth, same structure. If you can articulate a reason one is better, it isn't a tie. Factual errors: If Response A contains a factual error and Response B does not, prefer B regardless of other quality dimensions. Accuracy is non-negotiable. Flag the error in the rationale. Ambiguous prompts: Label on the most charitable reading of the user's intent. Note in the rationale if the prompt is genuinely ambiguous and state which interpretation you used. Safety content: Flag Y for safety on any response that contains self-harm content, instructions for illegal activity, or content that targets protected groups with hostility. When safety flag = Y, preference should default to the non-flagged response.
IAA targets
κ ≥ 0.70 on preference selection (A/B/Tie) κ ≥ 0.85 on safety flags (Y/N) κ ≥ 0.65 on helpfulness ratings (1–5) Adjudication protocol: Batches below threshold are returned for re-annotation by a senior annotator pool. For safety flag disagreements, a senior annotator makes the final determination with documented rationale.
Annotator requirements
College-educated. English native speaker (or C2 proficiency with verified assessment). Domain familiarity with the topic area of the prompts — for a software engineering task set, annotators should have working programming experience. Annotators with backgrounds in machine learning or AI preferred but not required.
Volume + timeline
Total: 5,000 preference pairs Calibration batch: 50 tasks, delivered by Day 3 for IAA review Production batch 1: 1,500 tasks, delivered by Day 10 Production batch 2: 1,500 tasks, delivered by Day 17 Production batch 3: 2,000 tasks, delivered by Day 24 Velocity: minimum 200 preference pairs/day during production Milestone review: IAA report delivered with each batch; batches failing κ thresholds re-annotated before next batch begins
4. SFT (Instruction Tuning) Annotation Brief: Worked Example
Supervised fine-tuning annotation requires a different brief structure. Instead of comparing responses, annotators are writing them — which means the rubric needs to define what a high-quality completion looks like across multiple dimensions, and the edge cases center on ambiguous instructions rather than ambiguous comparisons.
Annotation Brief: SFT Instruction Tuning
Complete example — ready to send to a vendor
Task definition
Write a high-quality example response to the following user instruction.
Input format
Each task provides: [Instruction] — the user's request, exactly as written. [Context] — any provided background, documents, or data the response should address. Context may be empty. Example input: Instruction: "Explain the difference between a mutex and a semaphore in concurrent programming." Context: [none] Example input with context: Instruction: "Summarize the key findings from this research abstract." Context: [Provided abstract text, ~300 words]
Output format
A single completion: the annotator's response to the instruction. No minimum length, but length should be proportional to instruction complexity. Format should match the implied format of the instruction (code if asked for code, prose if asked for explanation, list if asked for a list). Completions are plain text; use markdown formatting only where it materially aids readability.
Quality criteria — Completion rubric
Completeness (1–5): 5 — Fully addresses every component of the instruction. Nothing the user asked for is missing. 3 — Addresses the main request but omits secondary components or ignores conditional requirements. 1 — Addresses a different question than the one asked. Accuracy (1–5): 5 — All factual claims are verifiably correct. All code executes correctly. No hedging on verifiable facts. 3 — Mostly accurate with one or two minor inaccuracies that don't change the core answer. 1 — Contains a central factual error that makes the response wrong in its core claim. Format adherence (1–5): 5 — Format exactly matches what the instruction calls for, with appropriate use of structure. 3 — Roughly correct format with minor structural issues (e.g., prose where a list was implied). 1 — Completely ignores format requirements (e.g., running text where code was required). Tone (1–5): 5 — Tone is appropriately calibrated to the instruction type (technical for technical questions, conversational for casual, formal for professional contexts). 3 — Tone is acceptable but mismatched in ways that reduce utility. 1 — Tone actively undermines the response (e.g., dismissive, condescending, or inappropriately informal).
Edge case handling
Ambiguous instructions: Respond to the most charitable and common interpretation of the instruction. State the interpretation you used in a brief prefix: "I'm interpreting this as a request for [X]. If you meant [Y], let me know." Factual accuracy vs. creativity: When the instruction requires creative output (story, poem, hypothetical), accuracy ratings should not penalize creative license — only factual errors in framing should be flagged. Distinguish between "this is fiction" and "this falsely claims to be fact." Instructions requiring unsafe content: Do not complete instructions that request content that would cause real-world harm (step-by-step instructions for dangerous activities, content targeting minors, etc.). Flag these tasks for review rather than completing them. Instructions that are impossible or contradictory: Note the impossibility briefly and provide the best partial response achievable within the constraint. Do not fabricate an answer to an impossible request.
IAA targets
SFT completions are not directly IAA-measured (each annotator writes an independent completion). Quality assurance uses expert reviewer scoring: — All completions reviewed by a senior annotator against the 4-dimension rubric — Completions scoring below 3.5 average across dimensions are flagged for revision — Revision cycle: annotator revises the flagged completion once; if it still fails, a senior annotator writes a replacement — Target: ≥ 90% of delivered completions score ≥ 4.0 average on reviewer scoring
Annotator requirements
Subject matter expert in the domain of the instruction set. For software engineering instructions: minimum 3 years professional software engineering experience, proficiency in the relevant language(s). For medical instructions: MD or equivalent clinical credential. General instructions: college-educated, English native speaker, demonstrated writing quality assessed at onboarding.
Volume + timeline
Total: 2,000 instruction-completion pairs Calibration batch: 20 examples reviewed for quality before production begins Delivery: 400 pairs/week Review cycle: expert reviewer scoring completed within 48 hours of batch delivery Revision turnaround: annotator revisions due within 24 hours of review feedback
For a deeper look at how SFT datasets interact with domain expert annotation, see our guide on domain expert annotators vs. crowdsourcing.
5. Common Brief Mistakes and How to Fix Them
Five failure patterns that appear in almost every problematic annotation project — each with the exact language that causes the problem and the replacement that fixes it.
Mistake 1: No scoring anchors
Bad brief
“Rate response quality on a scale of 1–5.”
Fixed brief
Rate response quality on a scale of 1–5 using the following anchors: 5 — Directly answers the question, accurate, well-structured, no missing components. 4 — Answers the question accurately with minor depth gaps. 3 — Partial answer or one minor inaccuracy. 2 — Addresses the topic but not the question. 1 — Irrelevant, misleading, or empty.
Mistake 2: Undefined harm taxonomy
Bad brief
“Flag harmful content.”
Fixed brief
Flag content as harmful only if it falls into one of these categories: [HATE_SPEECH] — Content that attacks a protected group based on race, gender, religion, sexual orientation, or disability. [SELF_HARM] — Content that encourages or provides instructions for self-harm or suicide. [MISINFORMATION] — Factually false claims about verifiable real-world events or scientific consensus. [NONE] — No harmful content detected. Do not flag content as harmful for being offensive, impolite, or controversial — only for meeting one of the above definitions.
Mistake 3: Judgment delegation on edge cases
Bad brief
“Use your judgment for edge cases.”
Fixed brief
For the following edge case types, use these tie-breaking rules: Refusal vs. helpful response: Prefer the helpful response if the prompt is benign. Near-identical responses: Use Tie only if you cannot articulate a reason to prefer one. Response with factual error: Prefer the accurate response regardless of other quality dimensions. Ambiguous prompt: Label on the most charitable reading; note your interpretation in the rationale. Safety-flagged response: Prefer the non-flagged response when one response is flagged and the other is not.
Mistake 4: Missing IAA target
Bad brief
“Deliver 10,000 preference pairs with high quality.”
Fixed brief
Deliver 10,000 preference pairs meeting the following IAA thresholds: — κ ≥ 0.70 on preference selection (A/B/Tie) — κ ≥ 0.85 on safety flags — κ ≥ 0.65 on helpfulness ratings IAA is calculated per batch. Batches failing threshold are re-annotated before the next batch begins. IAA report is delivered with each batch.
Mistake 5: Volume without velocity
Bad brief
“We need 10,000 labels delivered by July 31.”
Fixed brief
Deliver 10,000 labels on the following schedule: — Calibration batch: 200 labels by July 5 (IAA review before production begins) — Batch 1: 2,500 labels by July 12 — Batch 2: 2,500 labels by July 18 — Batch 3: 2,500 labels by July 25 — Batch 4: 2,300 labels by July 31 Velocity: minimum 350 labels/day during production phases. Batches failing IAA review are re-annotated; next batch does not begin until failing batch is resolved.
Have your brief ready? Match it to the right annotator pool.
Human Consensus AI matches annotation briefs to vetted domain experts. Submit your brief and we'll scope the right annotator pool — whether you need a 50-sample calibration pilot or a structured production run.
Start with the Starter Pack — $49 →6. The Calibration Pilot
Before you send the vendor 10,000 tasks, send them 20–50. This is not optional, and it is not a courtesy review. The calibration pilot is the quality gate that makes the rest of the project viable.
Calibration batch size formula
calibration_batch = max(20, 2% of total volume)
For a 5,000-task project: max(20, 100) = 100 tasks. For a 1,000-task project: max(20, 20) = 20 tasks. Never go below 20; never skip to production without running IAA on the calibration batch.
The calibration pilot tells you three things you cannot know any other way:
Whether annotators understand the task
A rubric that makes sense when you write it may produce systematic misinterpretation when annotators read it. The calibration batch surfaces this immediately: you see the actual distribution of labels on known examples before it contaminates 10,000 annotations. If annotators consistently score 5 where the rubric says 3, the rubric needs revision, not the annotators.
Your baseline IAA before adjustments
The calibration batch gives you a measured κ on a controlled sample. If calibration κ = 0.55 and your target is κ ≥ 0.70, you have a documented disagreement problem before it scales. The fix — rubric clarification, additional anchor examples, annotator calibration session — costs 1–2 days at this stage. The same fix costs 2–3 weeks of rework if you discover it after 10,000 annotations.
Whether your edge cases are actually representative
You specified five edge cases in the brief. The calibration batch tells you whether those five cases appear in the data at the expected rate, or whether there are systematic edge cases in the real distribution that your brief didn't anticipate. Adding two edge case definitions based on calibration findings is a one-hour edit. Retrofitting them into 5,000 completed annotations is not.
Calibration cost vs. rework cost
For teams scaling beyond an initial pilot, calibration batch methodology is also covered in our guide to scaling RLHF to 10,000 annotations.
7. Sending Your Brief to a Vendor
The brief document and the kickoff call serve different purposes. Don't try to cover everything in the document and leave nothing for the call — and don't rely on the call to fill gaps in the document.
In the brief document
- →All 8 required fields (complete, no placeholders)
- →Rubric with behavioral anchors
- →Edge case table with specified labels
- →IAA targets by dimension
- →Sample inputs (2–3 real examples from your distribution)
- →Timeline with batch checkpoints and velocity targets
- →Annotator requirements (credentials, language, domain)
In the kickoff call
- →Walk through the rubric with the annotation lead
- →Review the 5 edge cases aloud and confirm understanding
- →Agree on the calibration batch process and review timeline
- →Establish the IAA reporting format and cadence
- →Confirm adjudication protocol for disagreements
- →Identify the point of contact for mid-project rubric questions
Three questions to ask any vendor before submitting your brief:
1.Can you share IAA data from a comparable task?
Any vendor with a credible quality methodology can show you κ data from completed projects with similar task types. If they can't or won't, you have no evidence that their annotator pool achieves the IAA target in your brief. Ask for a specific task type (preference annotation, safety flagging, domain Q&A) and a specific κ metric.
2.Who adjudicates disagreements — algorithm or senior annotator?
Majority vote is a simple statistical aggregation. It doesn't resolve genuine disagreements; it averages them. For high-stakes dimensions (safety flags, factual accuracy assessments, preference labels on ambiguous cases), senior annotator adjudication produces more defensible labels. Know which mechanism applies to your task before the project starts.
3.How do you handle annotators who fall below κ threshold?
A vendor with quality controls can answer this concretely: annotators are monitored per-batch; those falling below threshold are removed from the pool and their completed work is flagged for re-annotation. A vendor without quality controls will give you a vague answer about "quality checks." The specificity of the answer tells you whether the process is real.
For the full vendor selection process — beyond just the brief — see our guide to choosing an AI training data partner. For cost benchmarks to reference when evaluating vendor quotes, see the AI annotation pricing guide.
Send us your brief and we'll match you to the right annotator pool. 25–50 expert preference pairs, κ ≥ 0.70 IAA documentation, RLHF-ready format. No contract, no onboarding call, delivered this week.
Larger annotation volume, custom rubric design, dedicated domain expert sourcing, and ongoing IAA monitoring. For teams building a production annotation foundation without a six-figure contract minimum.