The Core Difference
Instruction tuning (supervised fine-tuning, SFT) teaches the model what a good response looks like. You provide a prompt, you provide an ideal completion, and you minimize cross-entropy loss on the completion tokens. The model learns from demonstrated behavior: here is what you should produce given this input. The signal is explicit, the supervision is direct, and the optimization target is a specific output. The fundamental weakness is that it requires you to know, in advance, what the ideal completion looks like.
RLHF teaches the model how well it's doing according to human preference. You generate multiple candidate responses, show them to human evaluators, collect comparisons or ratings, train a reward model on that preference signal, and optimize the policy to maximize expected reward via PPO or similar. The signal is comparative, the supervision is indirect, and the optimization target is a learned function of human judgment. The fundamental weakness is that it requires human evaluators to make consistent, meaningful quality distinctions across thousands of pairs — and the reward model generalizes both the signal and any noise in that judgment.
Both require human data. SFT requires demonstration data — a human writing or selecting the ideal completion. RLHF requires preference data — a human ranking which of several completions is better. These are different cognitive tasks with different annotator requirements, different IAA dynamics, and different failure modes when quality drops. Conflating them leads to using the wrong tool, collecting the wrong data, and diagnosing model failures incorrectly.
When to Use Instruction Tuning
The right moment for instruction tuning is when you can specify what a good output looks like without relying on a comparative judgment. Three scenarios dominate.
Bootstrapping a new task domain. When a base model lacks the behavioral template for a task — generating structured JSON from unstructured clinical records, producing code completions in a proprietary API, responding in a constrained persona — SFT gives you the fastest path to in-distribution behavior. The model doesn't need to learn what "good" is relative to "bad"; it needs to learn the form of the output. FLAN demonstrated this at scale: 62 fine-tuning tasks with instruction templates improved zero-shot generalization across model sizes with no preference signal required.
Teaching format and style adherence. If your output requirements are precise — schema-compliant JSON, a specific markdown structure, a code style matching a codebase convention — SFT enforces those constraints more reliably than RLHF. Reward engineering to capture structural correctness is expensive and brittle. SFT on demonstrations of correct structure is direct supervision on the target you actually care about.
Domain adaptation on specialized corpora. Adapting a general-purpose LLM to a narrow vertical where "good" is largely unambiguous — legal extraction, clinical coding, financial NER — benefits from SFT before anything else. Alpaca demonstrated that 52,000 high-quality instruction-following pairs were sufficient for strong general task alignment; for a narrow domain vertical, 1,000–5,000 curated domain examples with a capable base model can be sufficient. The InstructGPT SFT phase used roughly 13,000 demonstration pairs to establish baseline instruction-following behavior before any RLHF began — establishing the behavioral regime that preference optimization then refines.
SFT scales down well. When budget is tight, scope is narrow, and correct outputs are well-defined, an SFT dataset for a focused task can be collected in weeks at modest cost. A properly designed instruction dataset for a narrow vertical can get you to 85–90% of the quality ceiling you're targeting — at a fraction of the operational complexity of a full RLHF pipeline. For a deeper look at what high-quality instruction datasets require, see our post on LLM fine-tuning data and instruction dataset quality.
When to Use RLHF
RLHF belongs in the pipeline when you need to optimize for quality along dimensions that are genuinely hard to specify in a demonstration. Three scenarios are canonical.
Open-ended generation quality. If your task produces long-form responses where multiple answers could be correct — and the distinction between a good response and a great one is in reasoning depth, comprehensiveness, tone, or helpfulness — you can't write an ideal demonstration because the target isn't a specific string, it's a quality distribution. InstructGPT's RLHF phase collected roughly 33,000 pairwise comparisons after SFT; the reward model trained on those comparisons generalized quality distinctions to held-out prompts in ways that SFT demonstrations couldn't encode directly.
Alignment with subjective preferences. Llama 2 Chat used approximately 1 million preference pairs across helpfulness and safety dimensions. At that scale, the goal isn't to teach the model a fixed response template — it's to align the model's output distribution with human preference across a distribution of prompts the team couldn't anticipate individually. That problem requires preference learning, not demonstration learning. Anthropic's Claude models follow the same pattern: Constitutional AI uses a combination of RLHF and AI-generated preference data, but the preference signal throughout is comparative, not demonstrative.
Reducing harmful or misaligned outputs. When the alignment problem is primarily about what the model should not do — harmful content, deceptive outputs, confidently wrong claims in high-stakes domains — the signal you need is a human evaluator identifying when an output crosses a threshold. Pairwise comparisons that distinguish safer from less-safe responses build a reward model that penalizes harm in ways an SFT dataset can't encode without exhaustively enumerating every failure mode. RLHF generalizes to distribution shift in harmful inputs; SFT on refusal examples doesn't.
The data requirement for RLHF scales broadly: InstructGPT used approximately 50,000 preference pairs; Llama 2 used over 1 million. The right scale depends on task diversity, the quality of your base policy entering RLHF, and how broadly you need the reward model to generalize. But the floor is higher than most teams expect — a reward model trained on fewer than 10,000 pairs typically lacks the coverage to generalize robustly. For a step-by-step guide to building a preference dataset, see how to build an RLHF dataset from scratch.
The Practical Decision Framework
Five questions resolve the vast majority of SFT vs. RLHF decisions in production pipelines:
1. Do you have clear, ground-truth correct outputs? If a domain expert could write the ideal response and two other domain experts would confirm it's correct — use instruction tuning. This applies to structured extraction, code generation against a spec, classification with a defined taxonomy, and any task where "correct" has an objective answer.
2. Do you need to rank subjective quality between alternatives? If "good" can only be defined by showing a human multiple candidates and asking which is better — use RLHF. This applies to long-form generation quality, conversational helpfulness, summarization quality, and any task where the quality target is a preference distribution rather than a specification.
3. Are you adapting to a new domain with labeled examples? SFT first, RLHF after. The base model needs domain behavioral priors before preference optimization over domain-specific quality dimensions is meaningful. Running RLHF for medical question answering on a model with no clinical reasoning priors produces noisy reward model training and unstable PPO.
4. Are you trying to reduce harmful or misaligned outputs? RLHF. Safety fine-tuning with SFT requires enumerating failure modes exhaustively — which scales poorly. Preference-based reward modeling with safety-oriented comparison criteria generalizes to novel harmful inputs; SFT refusal examples don't.
5. Is budget or timeline the binding constraint, and is scope narrow? SFT only, to start. A well-scoped instruction dataset for a focused task can be collected in weeks at modest cost. A full RLHF pipeline — preference collection infrastructure, reward model training, PPO stability work, ongoing evaluation — is a multi-month undertaking. For a narrow task with clear correctness criteria, SFT often achieves 90% of the quality ceiling at 10% of the operational cost. Add RLHF when you've hit that ceiling and confirmed the remaining gap requires preference modeling.
Sequencing Them: The Standard Pipeline
Most production-grade aligned LLMs use both approaches in sequence, and the ordering is not arbitrary — it reflects engineering constraints that are easy to get wrong.
The standard 4-stage pipeline: (1) pre-train on a large corpus to establish world knowledge and language modeling capability; (2) SFT on instruction pairs to establish instruction-following behavior; (3) reward model training on human preference pairs; (4) PPO policy optimization against the reward model, with a KL penalty to prevent divergence from the SFT baseline. InstructGPT described this 3-stage alignment pipeline explicitly (stages 2–4 above), and it has become the production template for aligned LLM training.
The reason SFT precedes RLHF is not just historical convention — it's a practical stability requirement. A raw pretrained model entering RLHF before SFT produces unstable reward model training. The completions sampled during RLHF data collection are too diverse in form: some will be reasonable instruction-following attempts, others will be document continuations, task failures, or structural outputs that don't match the intended behavior at all. Human evaluators' pairwise comparisons on that distribution can't build a coherent preference signal — they're ranking apples against code outputs against refusals.
The SFT step narrows the output distribution to the right behavioral regime before preference optimization begins. Once the model reliably attempts instruction-following, human preference pairs can meaningfully rank the quality of those attempts — and the reward model can generalize that quality signal to new prompts. The KL penalty in PPO prevents the policy from optimizing the reward model so aggressively that it diverges from the SFT baseline, which is why the SFT checkpoint is a reference point throughout RLHF training, not just a prior step.
Practical implication: if you're running RLHF and seeing reward model instability or PPO policy collapse in early iterations, check whether your SFT dataset was adequate for the task. Under-trained SFT — too few examples, poor quality demonstrations, or mismatched domain coverage — is one of the most common causes of RLHF failure in production pipelines, and it's often misdiagnosed as a reward modeling or PPO hyperparameter problem.
Human Data Quality Requirements for Each
The human data requirements for SFT and RLHF have distinct failure modes — and both fail quietly before showing up as model behavior problems downstream.
SFT demonstration data needs high-agreement, single-best-answer quality. An annotator writing or selecting an ideal completion for a complex domain task — a clinical reasoning question, a legal interpretation, a financial analysis — needs to be domain-credentialed, not just a fluent English speaker. The inter-annotator agreement bar for SFT demonstrations is κ ≥ 0.7 for complex judgment tasks. Below that threshold, disagreement is signal that the "ideal completion" isn't actually unambiguous — your SFT dataset contains conflicting supervision, and the model will interpolate between different human judgments rather than learning a consistent behavior. For narrow technical tasks where domain expertise is required, the agreement bar is often higher because the space of correct answers is smaller; disagreement is a more reliable indicator of annotator error rather than genuine ambiguity.
RLHF preference data needs consistent pairwise raters who can rank nuanced quality dimensions — helpfulness, harmlessness, accuracy, reasoning depth — across thousands of pairs without quality drift. The IAA threshold for preference pairs is typically κ ≥ 0.65, slightly lower than SFT demonstrations because pairwise rankings tolerate more evaluator subjectivity in how they weight competing quality dimensions. But annotator calibration is more critical for RLHF than for SFT: a preference pair collected in week one from a well-calibrated annotator and a pair from the same annotator after 2,000 tasks of drift may produce conflicting training signal even if both meet a per-pair IAA threshold. Calibration intervals of every 500–750 tasks are standard in well-run RLHF annotation operations.
For technical and professional domains — medical, legal, financial, code — both SFT demonstrations and RLHF preference pairs require domain experts. Generalist annotators on domain-specific tasks consistently produce IAA in the 0.35–0.50 range; domain experts on the same tasks achieve 0.65–0.80. That gap doesn't close with better rubrics or longer annotation guidelines — it reflects that quality judgment in specialized domains requires the background knowledge to recognize quality, not just apply a checklist. Both SFT and RLHF pipelines fail predictably when this is ignored: SFT fails because demonstrations encode the annotator's surface-level pattern-matching rather than correct domain behavior, and RLHF fails because the reward model learns to optimize for signals that correlate with quality for generalists but not for domain experts. See our guide on building a preference dataset for RLHF for annotation design specifics.