1. Why Annotation Quality Is Hard to Measure
The core problem with annotation quality isn't that annotators make random errors. It's that they disagree systematically. Two annotators reading the same preference pair bring different priors about what "helpful" means, different interpretations of an ambiguous rubric, and different thresholds for what counts as a safety concern. That systematic disagreement doesn't average out at scale — it encodes noise directly into your reward model training signal.
There are two distinct failure modes, and they require different fixes.
Low IAA — annotators disagree frequently. This means either the rubric is ambiguous (annotators are applying different criteria) or the task is genuinely hard (reasonable annotators reach different conclusions for legitimate reasons). Both are diagnosable and fixable before production.
High IAA on the wrong label — annotators agree consistently, but they're consistently wrong. This is the harder failure mode. If your rubric anchors are miscalibrated and every annotator has internalized the same flawed interpretation, IAA looks healthy while systematic bias enters your dataset undetected. Ground-truth comparison (annotator vs. expert baseline) is the only way to catch this.
Treat IAA as a diagnostic tool, not a pass/fail gate. A single κ number tells you whether annotators agree — not why they disagree or whether they're right. Use it to surface problems early, by dimension, before they compound in production.
2. Cohen's Kappa: The Standard Metric
Cohen's kappa is the industry standard IAA metric for pairwise annotation. The formula is:
In plain English: P_o is the fraction of annotation items where two annotators actually agreed. P_e is how often they would agree purely by chance — calculated from each annotator's marginal label distribution. If Annotator A labels 60% of items as "preferred" and Annotator B labels 55% as "preferred," chance agreement on the "preferred" label is 0.60 × 0.55 = 0.33. Sum the chance agreement across all labels and you have P_e.
Kappa subtracts that baseline before scoring. Percent agreement ignores it, which is why percent agreement is misleading — two annotators who both default to the majority class look like they agree even when one is essentially random.
| κ Range | Interpretation |
|---|---|
| < 0.20 | Slight agreement — rubric is broken or task is undefined |
| 0.21 – 0.40 | Fair — significant ambiguity in rubric or annotator pool |
| 0.41 – 0.60 | Moderate — marginal for most production use cases |
| 0.61 – 0.80 | Substantial — meets the bar for most RLHF annotation tasks |
| > 0.80 | Almost perfect — required for high-stakes tasks (safety, medical) |
The HH-RLHF paper (Anthropic's Helpful and Harmless dataset) targets κ ≥ 0.70 for preference annotation — which puts it in the "substantial" band. That's why 0.70 is the industry reference threshold for RLHF preference annotation: not because it's arbitrary, but because it reflects the agreement level at which preference signal is reliable enough to train a reward model on.
Weighted vs. unweighted kappa. Standard (unweighted) Cohen's kappa treats all disagreements equally. If Annotator A rates a response "5 out of 5" and Annotator B rates it "1 out of 5," that counts the same as a disagreement between 4 and 5. For binary preference pairs (A preferred vs. B preferred), unweighted kappa is correct. For ordinal ratings — 1–5 helpfulness scales, severity ratings, quality rubrics — use weighted kappa, which penalizes larger disagreements more heavily. The math changes; the interpretation table above still applies.
3. Beyond Cohen's Kappa: Other IAA Metrics
Cohen's kappa was designed for exactly two annotators and nominal categories. Most production annotation pipelines don't fit that profile — you typically have 3+ annotators, ordinal scales, and missing data from annotators who skip difficult items. Three other metrics cover the gaps.
Krippendorff's Alpha
The most flexible IAA metric. It handles any number of annotators, missing data (annotators don't need to rate every item), and any scale type — nominal, ordinal, interval, or ratio. Alpha calculates the ratio of observed disagreement to expected disagreement under chance, and applies a distance function appropriate to the scale type. Use Krippendorff's alpha as your default metric when you have 3+ annotators on ordinal or interval scales — it's strictly more general than kappa and handles real-world annotation pipelines better.
Fleiss' Kappa
An extension of Cohen's kappa to multiple raters on nominal (unordered) categories. It assumes a fixed set of raters who each rate the same set of items, and it corrects for chance the same way Cohen's does — but across all raters simultaneously. Common in safety classification tasks where 3–5 annotators each label items as safe/unsafe/ambiguous and the categories have no inherent ordering. It doesn't handle ordinal scales as well as Krippendorff's alpha, but it's more widely supported in annotation tooling.
Percent Agreement
Percent agreement is the fraction of items where annotators agree, with no chance correction. It looks good but it's misleading. On a task where 90% of items are class A, two annotators who independently default to class A get 90% agreement trivially — regardless of whether they're paying any attention to the other 10%. Never use percent agreement as your primary IAA metric. It's only useful as a sanity check alongside kappa.
| Scenario | Use This Metric |
|---|---|
| 2 annotators, binary preference pairs | Cohen's kappa (unweighted) |
| 2 annotators, ordinal scale (1–5 ratings) | Cohen's kappa (weighted) |
| 3+ annotators, nominal categories | Fleiss' kappa |
| 3+ annotators, ordinal / interval / missing data | Krippendorff's alpha |
4. Calculating IAA in Practice
Three steps before any annotation reaches production.
(a) Define annotation dimensions separately
Don't calculate a composite IAA across your whole task. A preference dataset typically has multiple evaluation dimensions — preference, safety, formatting, instruction-following. Each dimension gets its own kappa. A composite hides the dimension that's failing. Your annotators might be at κ = 0.82 on safety and κ = 0.55 on formatting — a composite of 0.70 looks fine, but your formatting rubric is broken and you don't know it.
(b) Collect a calibration batch
Before production begins, have all annotators independently label a shared calibration batch. Sizing formula: max(20, 2% of total volume). For a 1,000-pair RLHF annotation run, that's 20 pairs minimum. For a 5,000-pair run, that's 100 pairs. The calibration batch should include a representative spread of difficulty — not just easy items where everyone agrees trivially.
(c) Calculate per-dimension IAA before proceeding
If any dimension falls below your target threshold, stop and fix the rubric before production begins. Proceeding with κ < 0.60 on a critical dimension is the most common source of rework in annotation pipelines. See the annotation brief template for how to structure rubric anchor definitions that close IAA gaps before calibration.
Ground truth vs. pairwise IAA. Pairwise IAA measures how often annotators agree with each other. Ground truth IAA measures how often annotators agree with an expert baseline. Ground truth IAA is more informative for catching systematic bias — because pairwise IAA can stay high even when everyone is consistently wrong together. Ground truth comparison is required for safety classification and medical annotation. For preference tasks without a single correct answer, pairwise IAA is sufficient.
Worked Example
3 annotators rate 50 preference pairs. Pairwise kappa by annotator pair: A-B = 0.74, A-C = 0.71, B-C = 0.78. Average κ = 0.743 — meets the 0.70 RLHF threshold.
Dimension breakdown:
- Preference κ = 0.74 — annotators are applying the preference rubric consistently. Proceed.
- Safety κ = 0.82 — strong agreement. Safety rubric is clear and well-anchored.
- Formatting κ = 0.61 — below a reasonable target of 0.70. Root cause: the formatting rubric uses subjective language ("well-structured") without behavioral anchors. Fix the rubric before production begins.
If you had calculated a single composite IAA = 0.72 and stopped there, the formatting problem would have entered production undetected.
5. IAA Targets by Task Type
IAA targets vary by task because the cost of annotation errors is asymmetric. Safety annotation errors have higher downstream cost than preference annotation errors. Tasks with genuine subjective disagreement (creative preference) have structurally lower achievable κ than tasks with objective correct answers. Use this table as your baseline — and adjust based on your model's risk profile. For a deeper treatment of how to calibrate these targets for a specific project, see the reward model evaluation guide.
| Task Type | Minimum κ | Target κ | Notes |
|---|---|---|---|
| RLHF preference annotation | 0.65 | 0.70+ | HH-RLHF paper target. Genuine preference disagreement makes >0.80 rare on hard pairs. |
| SFT instruction quality | 0.70 | 0.75+ | Quality rubrics are more objective than preference; higher κ is achievable with clear anchors. |
| Safety / harm classification | 0.80 | 0.85+ | False negatives have asymmetric cost. Demand higher κ and use ground truth comparison — not just pairwise. |
| Named entity recognition | 0.75 | 0.80+ | Boundary disagreements (span start/end) are the dominant source of κ drag. Use span-level weighted metrics. |
| Sentiment analysis | 0.70 | 0.75+ | Fine-grained sentiment (5-class) has structurally lower κ than binary positive/negative. |
| Image captioning quality | 0.65 | 0.70+ | Subjective quality judgments; use ordinal scale with weighted kappa. |
| Code correctness | 0.75 | 0.82+ | Requires domain expertise. Expert annotators produce significantly higher κ than generalist crowd on non-trivial code. |
| Medical / clinical annotation | 0.75 | 0.80+ | Use licensed clinicians. Expert adjudication required for ties — majority vote is insufficient for high-stakes clinical labels. |
6. What to Do When IAA Is Low
Low IAA has three root causes, each with a specific fix. Diagnose before applying a solution — the wrong fix wastes time.
Root Cause 1: Ambiguous Rubric
Symptom: IAA is low across annotators who are otherwise calibrated well on simpler tasks. The rubric uses subjective language without behavioral anchors.
Fix: Replace subjective criteria with behavioral anchors. Not "rate quality 1–5" — instead, define exactly what a 1 looks like (e.g., "response fails to answer the question asked; contains factual errors") and what a 5 looks like ("response directly addresses all sub-questions; cites sources; handles edge cases"). Each rubric level needs its own anchor. A well-anchored rubric reduces annotation interpretation variance — see the annotation brief template for the full structure.
Root Cause 2: Annotator Miscalibration
Symptom: Individual annotators have low pairwise IAA with each other, but the disagreements don't cluster around specific rubric criteria — they're spread across the distribution. The rubric is clear, but annotators are applying it differently.
Fix: Run a calibration session. Collect 5–10 deliberately ambiguous examples (items where the "right" label is debatable) and have annotators label them independently, then discuss disagreements explicitly. Document the consensus ruling for each item. These become your canonical edge cases — the rubric interpretations all annotators have internalized together. Repeat with new examples monthly for long-running projects.
Root Cause 3: Genuinely Hard Task
Symptom: IAA is low even after rubric clarification and calibration. Expert annotators with deep domain knowledge still disagree on a meaningful fraction of items. This is real task difficulty — not a rubric problem.
Three options: (a) Reduce scope — narrow the task definition to remove the genuinely ambiguous cases (if 20% of items cause 80% of disagreement, exclude those item types and handle them separately), (b) increase annotator expertise level — specialists disagree less on domain-specific tasks than generalists because expertise resolves rubric ambiguity that training can't, (c) add an adjudication layer — see below.
The adjudication layer. For high-stakes tasks where κ < 0.60 persists after rubric fixes, use 3-way annotation with expert tie-breaking rather than simple majority vote. Majority vote is sufficient when the task has a clear correct answer and the minority vote is typically a random error. Expert tie-breaking is required when the disagreement reflects a genuinely difficult judgment call — the kind where a subject-matter expert's ruling is more reliable than two generalists outvoting one. Safety classification and clinical annotation fall in this category.
Red flag: annotator drift. IAA starts high in the calibration batch, then degrades over the production run. Annotators are drifting from the rubric interpretation they agreed to at calibration — a common pattern in long projects where the calibration session was a one-time event. Fix: weekly calibration refreshes with 5 new examples from the production batch. Drift is invisible if you only measure IAA at the start of a project.
7. How Human Consensus AI Handles Annotation Quality
Our approach is built around one observation: majority vote aggregation hides systematic disagreement. If 2 out of 3 annotators agree, the minority vote is discarded — and the reasoning behind it is never surfaced. That discarded reasoning is often where rubric ambiguity lives.
We use a consensus-with-rationale methodology instead. Annotators must explain their reasoning, not just record their label. That requirement surfaces systematic disagreement before it becomes production noise — because two annotators who give the same label for different reasons are a rubric clarification waiting to happen. For domain expert annotators, the rationale also produces higher-signal training data: a reward model trained on preference labels with reasoning traces learns the decision boundary, not just the majority outcome.
Domain expert matching raises the IAA ceiling directly. Rubric ambiguity is resolved by expertise — an ML engineer annotating code generation outputs doesn't need the rubric to define what "correct" means, because they can evaluate correctness independently. For tasks where annotator expertise matters, κ increases when the annotator pool is matched to the task domain.
Built-in calibration is included in every project. IAA is calculated per-dimension before production begins, and tracked across the project lifecycle — not measured once at kickoff and forgotten. We surface dimension-level IAA in project reporting, so you can see the formatting κ separately from the preference κ separately from the safety κ, rather than getting a single aggregate that hides where the quality problems are.
If you want to validate our IAA methodology on a sample task before committing to a production run, that's exactly what the Starter Pack is designed for.