Why Multimodal AI Needs a Different Annotation Approach
A GPT-4V deployment produces the following response to an image of a crowded marketplace: “The image shows three people standing near a fruit stand with approximately six oranges visible on the left side.” The actual image contains nine people and no oranges — the round objects are decorative lanterns. The response is grammatically fluent, confidently stated, and completely wrong in two independent ways: wrong object classification, wrong count.
A standard text preference annotator presented with Response A (the above) versus Response B (a different plausible description) has no mechanism to detect either failure unless they also see and analyze the image. Even if they do see the image, distinguishing model hallucination from a genuine ambiguity in the image requires the kind of careful visual attention that general crowdworkers don't consistently apply.
This is the grounding problem in multimodal annotation. The annotator must evaluate two distinct things simultaneously: whether the model “saw” the image correctly (image understanding accuracy) and whether its response was accurate and useful given what it should have seen (response quality). These are not the same judgment, and collapsing them into a single preference label produces training signal that can't distinguish a fluent hallucination from a genuinely good response. Neither standard pairwise preference annotation nor automated metrics like BLEU or CIDEr resolve this — they operate on text alone.
For context on the broader annotation quality tradeoffs, see multimodal AI training data annotation.
What Multimodal RLHF Actually Involves
Multimodal RLHF for VLMs like GPT-4V, Claude 3, Gemini, and open-source architectures like LLaVA requires a three-layer annotation structure that text-only RLHF doesn't need.
Layer 1: Image understanding accuracy
Did the model correctly identify and describe what's in the image? This is a precondition judgment — not a quality judgment. An annotator evaluates whether the model's representation of the image matches ground truth: objects present, counts, spatial relationships, text content, contextual scene elements. A response that fails this layer cannot be evaluated on Layer 2 — its quality is contingent on a false premise.
Layer 2: Response quality given the image
Assuming the model correctly understood the image: is the response accurate, complete, grounded in the visual content, and at an appropriate level of detail for the task? This is where standard preference annotation criteria apply — but only after Layer 1 passes. The GPT-4V technical report's multimodal evaluation framework, Claude 3's image understanding benchmarks, and Gemini's multimodal capability assessments all decompose evaluation along similar lines: visual comprehension first, then response quality conditioned on comprehension.
Layer 3: Task-specific judgment
For OCR tasks: is the transcription character-exact, including punctuation and formatting? For medical imaging: is the interpretation safe-to-use by a qualified clinician — and if not, is the model appropriately deferring? For chart reading: are the extracted values and trends accurate to the source data? Task-specific criteria at Layer 3 are non-negotiable: a response that “sounds right” on a chest X-ray interpretation is not the same as a response that is clinically safe.
This three-layer structure means a single multimodal preference pair requires more annotator attention, domain knowledge, and calibration than a text-only pair at any comparable task difficulty. Budget and staffing assumptions from text RLHF do not transfer.
The Annotator Quality Problem
General crowdworkers consistently produce inter-annotator agreement (κ) below 0.40 on multimodal annotation tasks. Three specific failure modes drive this:
Hallucination detection failure
Crowdworkers can't reliably distinguish a model hallucination from an accurate description of a complex image. A fluent, confident response that describes an object that isn't there reads as plausible — especially to an annotator who isn't examining the image critically. Hallucination identification requires active visual attention and familiarity with common VLM failure patterns (object substitution, count errors, OCR misreads, spatial relationship inversion). Crowdworkers don't have this mental model.
Domain image incompetence
Domain-specific images — medical scans, legal documents, scientific charts, engineering schematics — cannot be accurately evaluated by annotators who don't have domain knowledge. An annotator who can't read a CT scan themselves cannot evaluate whether a VLM's description of that scan is correct. The annotation task inherits the domain expertise requirement of the image type. For domain-specific VLM use cases, domain expert annotators are mandatory, not a quality upgrade.
Text-in-image attention deficit
OCR and chart reading tasks require character-level attention to detail that crowdworkers systematically underperform on. Missing a decimal point in a data extraction task, misreading a digit in a table, or accepting an approximately-correct transcription as exact — these are the failure modes that make crowdsourced OCR annotation unreliable. The attention requirement here is closer to proofreading than preference rating.
The κ < 0.40 figure from crowdsourced multimodal annotation is not recoverable at scale — it reflects a fundamental mismatch between the task requirements and the annotator pool, not a calibration problem that more examples can fix. See domain expert annotators vs. crowdsourcing for AI training for the full analysis.
Need expert annotators for VLM training data?
The Starter Pack includes 500 expert-annotated preference pairs with documented IAA scores across text and visual domains — calibrated annotators, not crowdworkers.
View the Starter Pack →Domain-Specific Considerations
Multimodal annotation requirements differ substantially across image types. Treating all visual tasks as equivalent produces annotation quality that's reliable on none of them.
Natural image and scene understanding
Annotators for general natural images need calibration specifically on edge cases: partially occluded objects (where the model must infer from partial evidence), ambiguous scenes (multiple valid interpretations), and optical illusions (where the perceptual phenomenon itself is what the model should describe). Without calibration on these scenarios, annotators apply inconsistent judgment to the exact cases where VLMs fail most often. Standard calibration covers clear examples — but it's the edge cases that separate useful training signal from noise.
Document and chart understanding
Annotators must themselves be able to read the chart or document correctly before evaluating the model's response. An annotator who can't interpret a multivariate regression output table cannot evaluate whether the model's interpretation is accurate. Academic or technical background — matched to the document type — is required. “Can read charts” is not a sufficient qualification for scientific figure annotation; domain literacy is.
Medical imaging
Licensed clinicians only, matched to the imaging modality (radiologists for X-ray/CT/MRI, pathologists for histology, etc.). Safety disqualifiers apply unconditionally: no annotator should evaluate clinical accuracy of a medical image interpretation unless they hold the relevant clinical qualification. IAA requirements for safety-relevant dimensions are stricter than standard — κ ≥ 0.80 on safety flags is a minimum, not a target. Any annotation program that can't meet this threshold needs to reduce task complexity, not lower the IAA bar.
Code screenshots and UI
Software engineers with relevant language/framework experience, applying a correctness-first hierarchy: does the model's description of the code accurately reflect what the code does? Is a UI element identified correctly? Are any bugs or issues in the screenshot accurately noted? Style and idiomatic quality are secondary to correctness — the same hierarchy that applies in code generation RLHF applies to code screenshot annotation.
Practical Annotation Schema for Multimodal Preference Pairs
A production-ready schema for multimodal VLM preference pairs requires five components. Text-only preference schemas are missing the first and fourth.
Image metadata
Image type (natural scene / document / medical / code-UI), domain (e.g., clinical radiology / financial chart / Python UI), and complexity rating (low/medium/high). Complexity rating enables stratified sampling in calibration and downstream analysis of annotation difficulty distribution.
Prompt and question
The exact prompt given to the VLM, including any task framing ("describe this image", "answer the question about this chart", "transcribe the text"). Task framing determines which Layer 3 criteria apply.
Response A and Response B
The two model outputs being compared. Both responses must be presented alongside the image — not as standalone text. Annotators who evaluate without the image produce random signal on the grounding dimension.
Per-dimension ratings
Image grounding: correct / partial / incorrect. Factual accuracy: correct / partially-correct / incorrect. Completeness: complete / missing-details / incomplete. Safety flag: yes / no (task-dependent — mandatory for medical, legal, and any response making consequential claims). These dimensions are rated independently per response, not as relative preference.
Overall preference with rationale
Which response is preferred overall, and a free-text rationale (2–4 sentences minimum) that explains the judgment with reference to specific visual content. The rationale is not optional — it's the primary signal for diagnosing annotator errors during calibration review.
Multimodal annotation at this schema depth takes 4–6× longer than text-only preference annotation on equivalent tasks. A text pair that takes 4 minutes takes 16–24 minutes for a comparable multimodal pair with a complex image. Budget accordingly — per-task pricing that works for text annotation will systematically underpay for multimodal annotation and produce quality that reflects it. For scaling guidance, see scaling RLHF to 10,000+ annotations.
Calibration for Visual Tasks
Standard calibration round design for multimodal tasks is similar in structure to text-only calibration but requires more careful calibration set construction.
Calibration set size: 30–50 image-response pairs. This is larger than typical text-only calibration (often 20–30 pairs) because the failure mode space is wider. Focus the calibration set on your model's specific hallucination patterns — LLaVA architectures have characteristic failure modes on object counting and spatial relationships; GPT-4V has documented issues with OCR on stylized fonts and small text; Gemini's multimodal performance varies systematically across image resolution ranges. Your calibration set should include pairs that surface whatever your specific model gets wrong, not generic visual QA examples.
IAA targets: κ ≥ 0.70 on factual accuracy before annotation begins. κ ≥ 0.80 on safety flags — the stricter requirement reflects the higher cost of safety annotation errors. If your annotator pool can't reach κ ≥ 0.70 on factual accuracy after a calibration round and feedback session, the problem is either annotator selection (wrong domain expertise) or task specification (ambiguous rubric). Don't start data collection at κ = 0.58 planning to improve over time — the improvement rarely arrives, and the early data is unreliable.
⚠ Skip calibration
- κ ~0.35 on factual accuracy
- 40–60% of annotations unusable
- Hallucinations treated as correct descriptions
- No adjudication standard for disputes
✓ Calibrated pipeline
- κ ≥ 0.70 on factual accuracy
- κ ≥ 0.80 on safety flags
- Consistent hallucination detection
- Adjudication protocol defined upfront
Adjudication protocol for image grounding disputes: When two annotators disagree on whether a model correctly interpreted an image, the adjudication goes to a third expert who views the image independently — without seeing the other annotators' ratings or rationales. This prevents the third rater from anchoring on an existing position. The independent third view is the ground truth for that pair. This protocol is more conservative and more expensive than standard majority vote, but grounding disagreements usually reflect a genuine ambiguity in the image or annotator error that needs clean resolution — not a plurality judgment.
For the full annotator prompt design methodology that makes calibration effective, see how to write better prompts for RLHF annotators. And for reward model evaluation once you have training data, see how to evaluate RLHF reward models.
Getting Started: Minimum Viable Multimodal Dataset
Sizing: 1,000–3,000 preference pairs per visual domain. Natural images, documents/charts, medical images, and code/UI screenshots are distinct visual distributions — don't mix them into a single training set and expect a unified reward model to perform well on all of them. Each domain requires its own annotator pool, calibration set, and preference dataset. A medical imaging reward model trained on data that includes natural scene annotations will produce unreliable scores on medical inputs where the grounding requirements are completely different.
Quality over volume, applied specifically: 1,000 expert-annotated multimodal preference pairs with κ ≥ 0.72 on factual accuracy consistently outperform 10,000 crowdsourced pairs (κ ~0.35) in VLM RLHF reward model quality. This gap is wider for multimodal tasks than for text-only tasks because the crowdworker failure modes (hallucination detection, domain image incompetence, OCR attention) compound directly into RM training signal. At κ = 0.35, roughly 40–50% of the crowdsourced annotations are wrong enough to actively mislead RM training. Scaling from 10K to 100K bad pairs doesn't fix this — it amplifies it.
Start with a calibration pilot of 30–50 pairs before committing to a full annotation run. The pilot tells you whether your annotator pool can reach IAA targets on your specific image types with your specific model's outputs. If it can't, the fix is annotator selection or task specification — neither of which gets cheaper after you've collected 2,000 unusable pairs.