In early 2025, a Series B fintech startup shipped a document-understanding model they'd spent four months building. The VP of Product had personally demoed it to investors. The model parsed loan applications, extracted key fields, and flagged incomplete submissions — it looked sharp in testing.
Six weeks after launch, customer complaints forced a rollback. The model was systematically misclassifying debt-to-income ratios as valid when they exceeded regulatory thresholds. The bug wasn't in the model. It was in the training data. The AI training data vendor they'd hired — chosen largely on price — had used a generalist workforce with no background in consumer lending compliance. The annotators had learned the field vocabulary. They hadn't learned the rules. Eighteen months of model work had to be rebuilt from scratch.
The VP of Product, who had no ML background, said afterward: "We didn't know what questions to ask."
That's the real problem. And it's more common than the people selling annotation services want you to know.
Why Outsourced Annotation Keeps Failing Non-Technical Buyers
Annotation vendors are not optimized for your model quality. They're optimized for throughput — tasks completed per day, cost per label, utilization rate. Those are the metrics their business runs on, and they're the metrics they'll report to you if you don't ask for anything else.
The quality problem is invisible until it's too late. Unlike a software bug that surfaces in testing, annotation quality issues hide inside training data. The model trains. It evaluates reasonably well on held-out data. It ships. And then it fails in production in ways that are slow to diagnose — because the failure looks like model behavior, not data behavior.
Most buyers don't know the right questions to ask because annotation is a specialized ops discipline at the intersection of ML engineering and workforce management. If your team doesn't have that expertise in-house — and most Series A–C AI product teams don't — you end up evaluating vendors on the criteria you understand (price, turnaround, platform UX) instead of the criteria that determine annotation quality for AI.
This post gives you the questions. They're not technical. You can ask them in a vendor call with zero ML background.
The 6 Questions to Ask Any Annotation Vendor Before You Sign
#1How do you screen and verify your annotators' domain expertise?
Not 'do you have domain experts' — how do you verify them? The answer should be specific: credential checks against professional registries, work history verification, domain-specific screener tasks with scoring thresholds. 'Self-reported profiles' or 'community reputation scores' are not verification. Push for documentation of the process. If they can't describe it in concrete steps, their expert pool is a marketing claim.
#2Do you provide inter-annotator agreement (IAA) metrics, including kappa scores, per batch?
Inter-annotator agreement (IAA) measures how consistently multiple annotators label the same item. Fleiss' kappa is the standard metric for multi-annotator tasks. Expert annotators on domain-specific work should produce kappa scores in the 0.70–0.85 range. Crowdsourced generalist annotators typically land between 0.35–0.50. You want that number per batch — not an aggregate over all time — so you can track quality degradation before it compounds into your training run.
#3Can you handle the task formats we need — pairwise comparisons, scalar ratings, ranked outputs?
Don't assume. If you're doing RLHF preference data collection, you need pairwise comparison tasks (A vs. B). If you're scoring output quality, you need scalar rating interfaces. Some vendors only support basic classification tasks and will try to fit your work into their format. This creates data quality problems that are hard to diagnose downstream. Describe your exact task types and ask them to show you the interface.
#4Will you run a paid pilot on 50 of our actual tasks before we commit?
A legitimate annotation vendor will say yes. The pilot lets you evaluate output quality against a known-answer calibration set and proves the vendor can handle your specific task type before you commit budget. Any vendor that requires a minimum commitment before you can evaluate quality is protecting their close rate at your expense. Walk away.
#5What output formats do you support — can you deliver clean CSV or JSON?
Post-processing annotation output adds latency and introduces transformation bugs. You want clean, documented schemas in standard formats — CSV or JSON. Ask for a sample output file, not a format description. If they can't give you one, assume the integration will be painful and the AI data labeling cost will escalate.
#6What's your escalation path when annotators disagree on an edge case?
This is the question almost nobody asks, and it reveals the most about a vendor's quality culture. Edge cases are where annotation quality actually gets decided. The answer should include a defined escalation process: disagreements above a threshold trigger review by a senior annotator or subject-matter expert panel. 'Majority vote wins' is not an escalation process — it's a way to launder noise into your dataset.
Red Flags That Mean Walk Away
- Anonymized rater pools with no credential visibility — If you can't inspect annotator qualifications — even at the aggregate level — you have no way to verify domain matching actually happened. Anonymous pools default to availability, not expertise.
- No IAA metrics or vague answers about quality control — 'We have quality assurance processes' means nothing without specifics. No kappa scores means no quality accountability. When you outsource data labeling, accountability requires measurement.
- All-inclusive flat pricing with no per-task transparency — Flat-rate pricing with no per-task breakdown hides annotator pay rates, which are often the first thing cut when vendors compete on price. Low annotator pay attracts low-effort annotators.
- No pilot option — If they won't run a small pilot before you commit, they're not confident in their quality. Don't be either.
How Human Consensus AI Answers All Six Questions
If you're evaluating an AI data annotation service for the first time, here's how Human Consensus AI stacks up against each question:
- Domain expertise verification — Annotators are recruited and screened by credential — professional licenses, institutional affiliations, and domain-specific screener tasks with scoring thresholds. You can review annotator qualification summaries before a task batch begins.
- IAA and kappa reporting — Fleiss' kappa is calculated and reported per batch — not an aggregate over all time. You get the metric alongside every delivery so you can track quality across engagements.
- Task format flexibility — Supports pairwise comparisons, scalar ratings, ranked outputs, multi-class classification, and free-text rationale fields. If you're building RLHF preference data, the platform is designed specifically for that workflow.
- Mandatory pilot — Every new client engagement starts with a paid 50-task pilot on their actual task type. You evaluate quality before you commit to scale.
- CSV/JSON output standard — All deliveries export in clean, documented CSV and JSON schemas. Sample output files are available before you sign.
- Edge-case escalation — When annotators disagree beyond a defined threshold, the item escalates to a domain-expert panel review. Items without majority resolution are flagged and documented — not silently voted through.
If your current vendor can't match this on any of these six criteria, the quality gap is showing up in your training data whether you can see it yet or not.
The Takeaway
Outsourcing AI training data doesn't mean losing control of quality — it means choosing a partner who makes quality legible. Kappa scores per batch. Verified credentials. Defined escalation paths. These aren't advanced ML concepts. They're the same accountability standards you'd apply to any operations vendor.
The AI data labeling cost that hurts isn't the per-task rate. It's the cost of retraining on bad data six months from now. Ask the six questions before you sign. Require the pilot. And don't accept "trust us" as a quality metric.