The Ship That Shouldn't Have Shipped
Fourteen months ago, an ML team at a mid-size AI company was three weeks out from launching a customer-facing summarization model. They had been benchmarking throughout development. ROUGE-L on their test set: 0.47 — above the internal threshold. BERTScore F1: 0.89. Perplexity on held-out data: competitive with their baseline. Every automated metric they tracked said the model was ready.
They shipped. Within 72 hours, support tickets flagged a pattern: the model produced summaries that were grammatically clean and factually adjacent to the source — but consistently missed the main point. Summaries of financial reports buried the key figure in paragraph three. Summaries of support conversations identified the wrong issue as the central complaint. The model had learned to produce text that matched surface-level statistical properties of "good summaries" without learning what a summary is actually for.
They pulled the feature, ran a human evaluation study — something they hadn't done before launch — and discovered a 34% rate of "missed main point" errors, invisible to every automated metric they had used. Four weeks of re-evaluation, partial retraining, and a delayed relaunch.
This is not an edge case. It is the predictable consequence of treating automated metrics as a proxy for what users actually care about. BLEU, ROUGE, BERTScore, and perplexity measure specific, narrow properties. None of them measure whether a model output is useful to a real human in a real context. That gap is why human evaluation for LLMs is not optional — it is the only ground truth that matters.
Section 1: The Automated Metrics Trap
Automated evaluation metrics were designed for specific, constrained tasks — and most of them predate modern language models by years. BLEU was developed for machine translation in 2002. ROUGE for summarization in 2004. They measure n-gram overlap between a model output and a reference string. They were never designed to evaluate whether a response is good — only whether it looks like a known-good example.
The failure modes are well-documented and consistent:
BLEU and ROUGE: Surface similarity without meaning
Consider two model summaries of a news article about a company missing earnings:
Reference: "Acme Corp reported Q3 revenue of $2.1B, falling 18% short of analyst estimates. CEO cited supply chain disruptions."
Model A: "Acme Corp reported Q3 revenue and analyst estimates for the quarter, citing disruptions in supply chain operations across the company."
Model B: "Acme Corp had a strong Q3, with revenue in line with expectations. CEO expressed optimism about the outlook."
Model A scores higher on ROUGE — it shares more n-grams with the reference. But it omits the key numbers entirely. Model B is factually wrong. Neither failure registers as a failure in automated scoring. A human evaluator flags both immediately.
BERTScore: Semantic similarity, not factual correctness
BERTScore improves on BLEU by using embedding similarity rather than exact token matching. It's better at capturing paraphrase and semantic relatedness — but semantic similarity and factual correctness are not the same thing. A response can be highly semantically related to a reference while containing a critical factual error, a contradicted claim, or a dangerous omission. BERTScore has no mechanism for detecting any of these.
Perplexity: Fluency without substance
Perplexity measures how well a language model predicts a token sequence — lower is better. It's a proxy for fluency and coherence. A model that generates grammatically perfect, confidently stated nonsense will score well on perplexity. A model that produces correct but awkward technical explanations may score worse. Perplexity has essentially no correlation with helpfulness, factual accuracy, or task completion on open-ended generation tasks.
The common thread: automated metrics measure statistical properties that correlate with quality in specific, constrained conditions and become progressively less reliable as tasks become more open-ended, subjective, or domain-specific. For modern LLM applications — instruction following, multi-turn dialogue, domain-expert tasks, safety-critical generation — the correlation breaks down almost entirely.
Section 2: What Human Evaluation Actually Measures
When researchers say human evaluation is the gold standard, they don't mean asking annotators "is this good?" on a 5-point scale. They mean structured evaluation across multiple dimensions that automated metrics can't touch. Here are the five that matter most:
1. Factual Correctness
Does the output contain accurate information? Are stated facts verifiable? Are there unsupported claims, hallucinated figures, or confident errors? This requires a human — often a domain expert — who can actually evaluate whether what the model says is true. Automated metrics have no access to the world and no way to make this assessment.
2. Helpfulness / Task Completion
Did the model actually accomplish what the user was trying to do? This requires understanding user intent — which is not inferable from the text of the response alone. A model can produce a long, well-structured answer that completely fails to address what was asked. Human evaluators with context about the task can catch this; automated metrics cannot.
3. Fluency and Naturalness
Does the response read the way a competent human would write it? Modern LLMs score reasonably well on perplexity, but fluency as humans experience it involves subtle judgments — appropriate register, natural transitions, absence of awkward constructions — that native speaker evaluators catch in ways that token-level metrics don't capture.
4. Safety and Alignment
Is the response safe? Does it contain harmful content, biased framing, or dangerous advice? Safety evaluation is fundamentally context-dependent — the same sentence can be benign in one framing and harmful in another. Human evaluators with training in harm taxonomy are the only reliable approach for this dimension. Classifier-based safety filters are a coarse proxy that misses novel harm patterns and context-dependent risk.
5. Domain Appropriateness
Does the output meet the standards of the domain it's operating in? A response that's superficially correct about a clinical topic but uses imprecise terminology, misrepresents standard of care, or fails to flag important caveats may be worse than no response at all. Only evaluators with actual domain expertise can make this judgment — and it's the dimension that most general annotation platforms are structurally unable to assess.
Section 3: Human Evaluation Done Wrong
The argument for human evaluation of language models is not an argument for any human evaluation. Poorly designed human eval studies produce unreliable data that is sometimes worse than a good automated metric. The failure modes are specific:
- Inconsistent raters with no IAA tracking. If inter-annotator agreement isn't measured, you don't know whether your labels reflect anything real. A study with 3 raters and no IAA reporting is producing anecdotes, not evaluation data. Baseline: target Cohen's κ > 0.70 for subjective dimensions, κ > 0.80 for factual correctness tasks.
- Annotation fatigue destroying calibration. Raters asked to evaluate 200+ responses in a single session show measurable quality degradation after the first 60–80 items. Fatigue manifests as increased central-tendency bias (everyone gets 3s), faster response times, and lower agreement with co-raters. Sessions should be capped and attention checks embedded.
- Domain novices rating domain-expert tasks. This is the most expensive mistake in human eval. General crowd annotators can reliably assess fluency and tone. They cannot reliably assess factual correctness in medicine, law, finance, or technical domains. Deploying a crowd to evaluate a medical LLM on clinical accuracy produces labels that feel rigorous but are essentially noise — and models trained on that signal inherit the confusion.
- Poorly designed rating tasks and ambiguous rubrics. "Rate this response on a scale of 1–5 for quality" is not an evaluation task. Different raters will interpret "quality" differently, and the resulting scores are not comparable across raters or studies. Effective eval tasks specify exactly what to evaluate and what each rating level means in observable terms.
Each of these failure modes creates demand for what general annotation platforms typically can't provide: structured expert panels, domain-credential vetting, task-specific calibration, and IAA monitoring at scale. Getting AI model evaluation best practices right requires building the eval study the same way you'd build a research study — with attention to rater selection, task design, and reliability from the start.
Section 4: Automated vs. Human vs. Hybrid — A Practical Framework
The goal isn't to replace automated metrics entirely — it's to use each approach for what it's actually good at.
| Dimension | Automated | Human | Hybrid |
|---|---|---|---|
| Cost | Very low — runs in seconds | Higher — scales with task complexity and rater expertise | Moderate — auto filters reduce human eval volume |
| Speed | Near-instant — suitable for CI/CD integration | Days to weeks depending on panel size and volume | Faster than full human eval; slower than pure auto |
| Accuracy: Surface-level tasks | High — n-gram overlap, format compliance, length | High but wasteful — overkill for surface checks | Auto handles surface; human reserves bandwidth for nuance |
| Accuracy: Nuanced tasks | Low — can't assess helpfulness, intent, or domain correctness | High — only approach that captures task-level quality | High — human eval on targeted sample with auto pre-screening |
| Scalability | Scales infinitely — run on any dataset size | Constrained by rater availability and domain expert supply | Scales well — humans review a targeted subset |
| Reliability: Safety-critical evals | Low — classifiers miss novel harm patterns and context | High — context-aware judgment with trained raters | High — classifiers flag candidates, humans make final call |
| Reproducibility | Perfect — same inputs always produce same scores | Variable — requires IAA controls to maintain reliability | Good — auto component reproducible; human component controlled |
The practical framework: use automated metrics for regression testing and CI/CD gates where you need fast feedback on whether something broke. Use human evaluation for any quality judgment that involves helpfulness, accuracy, safety, or domain appropriateness. Use hybrid approaches for large-scale studies where you need to triage efficiently — auto metrics identify obvious failures, human evaluators assess the remaining population for dimensions that matter.
The one category that should never rely on automated-only evaluation: safety-critical applications. If your model operates in medical, legal, financial, or high-stakes decision-support contexts, classifier-based safety filtering is not sufficient. The failure modes of safety classifiers — false negatives on novel harm patterns, inability to model context — are exactly the failure modes that matter most when the stakes are highest.
Section 5: How to Structure a Human Eval Study
A well-designed LLM human evaluation study has five components. Each one is a lever you can adjust — and each failure mode has a specific effect on data quality.
1. Rater Selection Criteria
Match rater credentials to the evaluation task. Fluency and naturalness can be assessed by native speakers without domain expertise. Factual correctness requires domain knowledge. Safety evaluation requires trained raters who understand harm taxonomy. Domain appropriateness requires genuine practitioners. Define the minimum credential bar before recruiting — not after you've collected labels and can't explain the variance.
2. Task Design and Rubric Specificity
Each evaluation dimension should have a task that isolates it. Factual correctness: "Based on the provided reference document, identify any factual errors or unsupported claims in the model output." Helpfulness: "Given this user request, rate how completely the response addresses what was asked." Rubric anchors should use observable criteria, not subjective adjectives. "1 — Response does not address the user's question at all" is an anchor. "1 — Very unhelpful" is not.
3. Inter-Annotator Agreement Targets
Set IAA targets before the study, not after. Guidelines: κ > 0.80 for factual correctness; κ > 0.70 for helpfulness and fluency; κ > 0.85 for safety evaluation. Run a calibration batch of 50–100 items before production annotation. If IAA falls below target in calibration, diagnose whether the issue is rater selection, guideline ambiguity, or task design — and fix it before scaling.
4. Sample Size and Statistical Power
For comparing two models or two fine-tuning runs, you need enough samples to detect meaningful differences. A common mistake: running 50-item human eval studies and drawing conclusions from differences that aren't statistically significant. Minimum recommended sample: 200 items per condition for detecting moderate effect sizes. Use 3 raters per item for dimensions where variance is expected, and resolve disagreements through adjudication rather than averaging.
5. Quality Gates and Attention Checks
Embed attention checks — items with known-correct answers — throughout the evaluation batch to monitor rater quality in real time. Set a threshold for exclusion: if a rater fails more than 15% of attention checks, their labels should be reviewed before inclusion. Track per-rater IAA over time — quality degradation during a session is a signal to pause and recalibrate, not a reason to discard data after collection.