1. Why This Debate Is Happening Now
The "synthetic data solves everything" narrative got real ammunition starting around 2023. Phi-1 and Phi-2 from Microsoft demonstrated that models trained on textbook-quality synthetic data punched far above their parameter weight. WizardLM showed that Evol-Instruct — a method for synthetically evolving instruction complexity — could match or beat models trained on curated human datasets. Then Llama 3's Magpie methodology showed that a capable base model could generate 300,000+ high-quality instruction pairs by prompting itself, with results competitive with expensively curated human data on benchmark tasks.
The logical extrapolation — "just generate all the data you need" — was irresistible and partly correct. For SFT on well-understood tasks, synthetic generation at scale is the right call. The counterforce arrived from the alignment side: RLHF research consistently shows that reward model quality equals human preference data quality, not LLM preference data quality. DPO papers show the same. Constitutional AI reduces but does not eliminate the human expert requirement. Every major alignment approach still requires human input at the stages that matter most for actual behavior alignment.
The debate isn't resolved — it's task-specific. Knowing which tasks belong in which column is the engineering judgment that determines whether your training pipeline produces a model that actually behaves the way you intend.
2. What Synthetic Data Is Good At
Be honest about this. Synthetic data has four real advantages that matter for production training pipelines — not theoretical ones.
Instruction diversity at scale
Magpie-style self-instruct generates 300,000+ instruction-response pairs in hours. Human annotation at that volume isn't just expensive — it's infeasible on any reasonable timeline. For tasks where diversity of instruction surface matters more than any individual pair's quality, synthetic generation is the only practical path. A human-annotated dataset of 5,000 pairs will have better individual quality; a synthetic dataset of 300,000 pairs will have better coverage of the instruction distribution.
Domain coverage expansion from seed data
Seed 50 high-quality human expert examples, then synthesize 5,000 variations around them. You get 100× coverage expansion without 100× human annotation cost. The synthetic variants inherit the domain framing and quality bar of the seed examples while diversifying the instruction surface, output format, and complexity level. This is the right hybrid: human experts define the quality ceiling, synthetic generation expands the coverage floor.
Format consistency with zero annotation variance
Synthetic generation produces perfectly structured JSON, markdown, and code examples with no annotation variance — because there's no annotator. For tasks where output format consistency is a training objective (structured extraction, code generation, API-compliant responses), synthetic data eliminates the format noise that comes from asking 20 different humans to produce JSON objects.
Cold-start bootstrapping before human annotation
When no labeled data exists for a new domain, synthetic generation gives the first fine-tuning signal before human annotation begins. A model fine-tuned on synthetic warm-start data will produce higher-quality outputs that are easier to annotate — better calibrated questions for RLHF, more clearly differentiated response pairs for DPO. The human annotation budget goes further when applied to a model that already has a reasonable initial distribution.
Where synthetic genuinely fits
SFT on well-understood tasks: code generation, summarization, translation, mathematical reasoning, structured output formatting. These tasks have clear rubrics, and LLM judges reliably evaluate them. The generator model has sufficient knowledge to produce correct examples. Synthetic is the correct choice at scale for these use cases.
3. Where Synthetic Data Breaks Down
Five specific failure modes. These aren't edge cases — they cover the most consequential stages of alignment and fine-tuning work.
Sycophancy inheritance
LLM-generated preference data inherits the generator model's stylistic biases. If GPT-4 prefers verbose, formally structured responses, your reward model learns to prefer verbose, formally structured responses — regardless of whether actual humans prefer them. The self-referential feedback loop is the structural problem: the generator's priors become the reward model's priors, which become the policy's behavior. You don't discover this misalignment until you deploy and real users find the outputs worse, not better, than the baseline.
Novel capability ceiling
Synthetic data can only recombine what the generator model already knows. It cannot introduce genuinely new domain knowledge, novel failure modes, or specialized expertise that wasn't in the generator's training data. If you're fine-tuning for a highly specialized medical subspecialty, frontier legal reasoning, or cutting-edge chemistry synthesis, the LLM generator is pattern-matching to what authoritative text in those domains looks like — not generating from domain expertise. The resulting data has the shape of expertise without the substance.
Safety and harm annotation failure
Harm is contextually and culturally situated. LLM judges systematically miscalibrate on domain-specific harm assessment: medical misinformation at the knowledge frontier, legal advice in non-US jurisdictions, culturally-specific content moderation. A concrete data point: GPT-4 agreement (κ) with expert human annotators on medical harm classification runs approximately 0.55 — below the 0.70 threshold for reliable training signal. Using LLM-generated harm labels at this agreement level adds noise to your safety classifier, not signal. The label distribution looks reasonable; the individual labels are systematically wrong in ways that correlate with what the LLM doesn't know.
Preference alignment ground truth
This is the core failure mode for RLHF and DPO. Both methods require preference pairs that reflect human preferences. Using LLM-generated preference pairs trains the model to be preferred by LLMs, not by humans. Those are different objectives. A reward model trained on LLM-generated preferences is optimizing for a proxy — "what does GPT-4 prefer?" — not the actual target: "what do the humans using this model prefer?" The misalignment between proxy and target grows with every RLHF iteration. For DPO specifically, this problem is more severe: without an intermediate reward model to absorb and smooth annotation noise, low-quality preference pairs bake directly into the policy weights.
Regulatory compliance
GDPR, HIPAA, and EU AI Act Article 14 require human oversight in high-stakes automated decision systems. Fully synthetic annotation pipelines don't produce the human review audit trail that compliance requires. For medical, legal, financial, and other regulated deployment contexts, the question isn't whether synthetic annotation is cheaper — it's whether it's permissible. In most regulated contexts, it isn't as a standalone path. See the RLAIF vs. RLHF post for a detailed breakdown of the compliance gap.
4. The Training Stage Decision Map
The pattern is consistent: synthetic scales the stages where the rubric is clear and the generator has sufficient knowledge. Human data is required where the rubric is ambiguous, the domain is specialized, or the stakes are high enough that calibrated human judgment is the only defensible source.
| Training Stage | Data Type | Recommended Source | Why |
|---|---|---|---|
| Pre-training | General knowledge | Web scrape + synthetic augmentation | Volume and diversity matter more than individual quality at this stage |
| SFT — well-understood tasks | Instruction-response pairs | Synthetic (Magpie, self-instruct) | Well-defined rubric; LLM judge reliably evaluates code, math, summarization |
| SFT — domain-specific tasks | Expert instruction-response | Human domain experts | Knowledge frontier; LLM can't generate what it doesn't know |
| RLHF preference data | Human preference pairs | Human annotators | Reward model quality = human feedback quality; LLM preferences ≠ human preferences |
| DPO preference data | Binary preference pairs | Human annotators with κ ≥ 0.70 | No reward model to catch noise — quality bar is higher, not lower |
| Safety/harm classification | Harm labels + severity | Human domain experts with κ ≥ 0.85 | Contextual harm requires human judgment; LLM-human κ ≈ 0.55 on medical harm — below threshold |
| Red-team adversarial | Adversarial prompts | Human red-teamers | Novel attack vectors require human creativity; LLM generates known jailbreak patterns, not novel ones |
The pattern: synthetic scales the stages where the rubric is clear and the generator has sufficient knowledge. Human data is required where the rubric is ambiguous, the domain is specialized, or the stakes are high enough that calibrated human judgment is the only defensible source.
Two rows deserve extra attention. DPO preference data: the conventional wisdom that DPO is simpler than RLHF leads teams to underestimate its annotation requirements. Without a reward model in the loop, a low-κ preference pair bakes directly into the policy — there's no intermediate diagnostic artifact to catch the error. The quality bar is higher, not lower. Safety/harm classification: the κ ≥ 0.85 target (vs. 0.70 for preference tasks) reflects the asymmetric cost of miscalibration. A wrong preference label makes your model slightly less aligned; a wrong harm label makes your safety classifier miss the thing it was built to catch. Read the preference dataset construction guide for how to structure both DPO and RLHF datasets to enforce the quality bar before training begins.
5. The Hybrid Pipeline Architecture
In practice, the question is never "synthetic or human for everything" — it's how to structure the pipeline so each data source is used where it has the best cost-quality tradeoff. A three-tier architecture covers most production use cases.
Tier 1 — High-volume, clear-rubric: synthetic with automatic filtering
Synthetic generation → automatic quality filtering (perplexity thresholds, format validation, length distribution checks) → use directly for SFT cold-start. No human in the loop. Appropriate when: the task is well-understood (code, math, summarization), the rubric is unambiguous, and the generator model has domain-sufficient knowledge. Cost: effectively zero per pair beyond compute.
Tier 2 — Moderate complexity: synthetic with human QA sampling
Synthetic generation → human spot-check on 5–10% sample → measure IAA on the sample → if sample passes κ ≥ 0.70, use full synthetic batch at scale. Human reviewers aren't annotating the full batch; they're running a quality gate on the synthetic generation process. If the sample fails, the full batch is flagged before it enters training. This is the correct architecture for moderate-complexity tasks where synthetic is likely sufficient but you need a quality signal to confirm it.
Tier 3 — High-stakes, domain-specific, preference: human annotation only
Human annotation → per-dimension IAA measurement → production. No synthetic data in the loop. Appropriate for: RLHF/DPO preference pairs, safety/harm classification, domain-expert evaluation, regulated deployment contexts, adversarial prompt generation. These aren't overflow from Tier 2 — they're tasks where Tier 1 would produce systematically wrong labels regardless of how well the automatic filters work.
Decision rule for tier assignment
What is the cost of a wrong label? Low (model slightly off-topic) → Tier 1 or 2. High (reward model misaligned, safety classifier fails) → Tier 3.
Is the error reversible? Can retrain next week → Tier 1 or 2. Deployed in production or regulated context → Tier 3.
Does the generator have domain knowledge? Yes → Tier 1 or 2. No (specialized domain) → Tier 3.
Most teams discover this architecture the hard way: start at Tier 1 because it's cheap and fast, discover quality problems downstream at the preference or safety stage, and then invest in human data at the specific stages where synthetic failed. The smarter path is to map the failure modes in advance using the decision table in the previous section. See the annotation pilot checklist to operationalize the hybrid architecture — including how to run the Tier 2 sampling process and what IAA targets to set before scaling.
6. Quality Measurement Across Both Sources
The metric that matters for evaluating whether synthetic data is reliable on your specific task is human-AI agreement (κ) — not self-consistency, which is high by definition for any deterministic generator. An LLM judge that produces 0.95 self-consistency can still be 0.55 against domain expert human annotators on specialized tasks. The high self-consistency score is evidence that the LLM is consistent with itself, not evidence that it's correct on your task.
Measuring human-AI κ in practice
- Run 50–100 items through both synthetic generation and human annotation in parallel. Use your production rubric, not a simplified version.
- Compute κ per dimension, not as a composite. Composite IAA scores hide per-dimension failures — a 0.78 composite can contain a 0.58 dimension that will add noise to your training signal.
- If κ ≥ 0.70 on a dimension, synthetic is viable for that dimension at scale. If κ < 0.70, synthetic is adding noise — not signal — for that dimension specifically.
For preference tasks specifically, run preference pairs through both your LLM judge and a human panel. Compare the rankings. If top-K agreement between LLM judge and human panel exceeds 85%, synthetic preferences are usable for SFT. If agreement falls below 70%, synthetic preferences will misalign the reward model — the judge's ranking doesn't reflect the humans' ranking closely enough to train toward human preferences.
This κ calibration exercise also tells you which tasks to route to which tier in your hybrid pipeline. High human-AI κ → Tier 1 or 2. Low human-AI κ → Tier 3. The IAA and kappa methodology post covers the full measurement approach — calibration batch sizing, weighted vs. unweighted kappa for preference pairs, and how to interpret κ by task type with the appropriate thresholds.
One more thing: the κ threshold differences between task types are not arbitrary. The 0.85 target for safety/harm vs. 0.70 for preference annotation reflects the asymmetric cost of getting it wrong. Low annotation agreement on a preference pair produces a slightly less aligned model. Low annotation agreement on a harm label produces a safety classifier that misses the thing it was built to catch. The domain experts vs. crowdsourcing analysis covers why expert annotators systematically achieve higher κ on specialized tasks — and why the expertise requirement doesn't go away when you switch from crowdsourcing to synthetic generation.
7. Where Human Consensus AI Fits (and Doesn't)
Honest fit assessment. Synthetic data is the right call for a significant portion of any production training pipeline. Human Consensus AI is not the right call for the parts where synthetic is sufficient.
Right for:
- RLHF/DPO preference pairs requiring human κ ≥ 0.70 — preference data that reflects what humans prefer, not what LLMs prefer. The reward model quality ceiling is set here. Both RLHF and DPO pipelines require this; DPO more stringently.
- Domain-specific SFT where LLM knowledge is insufficient — medical, legal, financial, scientific domains where the generator model is pattern-matching plausibility rather than domain expertise. Expert annotators evaluate correctness; LLM judges evaluate plausibility. These are different.
- Safety/harm annotation at κ ≥ 0.85 — per-dimension harm classification with documented inter-annotator agreement and rationale capture. Contractual κ targets for regulated deployment contexts.
- Red-team adversarial prompt generation — novel attack vectors require human creativity. LLM-generated adversarial prompts cluster around jailbreaks in training data. Human red-teamers find attacks that feel human because they are human.
- Hybrid pipeline Tier 3 and Tier 2 QA — both direct Tier 3 annotation and the human QA sampling layer that validates Tier 2 synthetic batches before they enter training.
- Regulated deployment with audit trail requirements — HIPAA, GDPR, EU AI Act Art. 14, FDA SaMD contexts where AI-only annotation pipelines create compliance gaps.
Not right for:
- SFT on well-understood tasks at 100K+ scale — if human-AI κ calibration shows ≥ 0.80 agreement on your task, synthetic generation is the right call. No reason to pay expert annotation rates for pairs that LLMs label correctly.
- Pre-training data augmentation — volume and diversity matter more than individual quality at pre-training scale. Synthetic augmentation is the appropriate tool.
- Automated QA pipelines where LLM judge is calibrated — if you've run the κ calibration and human-AI agreement is ≥ 0.80 on your specific task, the automated pipeline is sufficient.
The key insight: Synthetic data doesn't replace human feedback — it frees up the human annotation budget to focus on the stages where it creates irreplaceable value. The teams that use both well spend their human annotation budget on RLHF preference pairs, domain-expert SFT, and safety classification. They use synthetic generation for everything else. The hybrid pipeline isn't a compromise — it's the architecture that gets you the most value from each source.