Constitutional AI and RLHF: How Anthropic Uses Human Feedback for AI Safety

The Scaling Problem That Motivated Constitutional AI

Pure RLHF has a well-documented bottleneck: it scales linearly with human annotators. Every preference pair requires a person to read two model outputs, apply a rubric, and record a judgment. At the scale Anthropic, OpenAI, and DeepMind are training their frontier models, that pipeline becomes expensive, slow, and — more importantly — inconsistent. OpenAI's InstructGPT paper reported that annotator agreement on helpfulness ratings was approximately 72.6% on average. Across a dataset of tens of thousands of pairs, the noise floor is substantial.

But cost and throughput aren't the deepest problem. The deeper problem is that human annotators are not safety experts. When you ask a crowd worker to rate whether a model response is "harmful," they're applying a naive harm intuition that doesn't distinguish between surface-level offensiveness and genuinely dangerous content — between a response that mentions violence in a historical context and one that provides operational details for it. At scale, those distinctions matter enormously, and general annotators make them inconsistently.

Anthropic's insight with Constitutional AI was to decouple the two bottlenecks: volume and quality. The volume problem — how to generate and rate enormous numbers of preference signals — could be addressed with AI-generated feedback. The quality problem — how to define the boundaries of acceptable behavior with precision — could not. That part still requires human judgment.

What Constitutional AI Actually Is

Constitutional AI (CAI), introduced in Anthropic's 2022 paper, is a two-phase training approach designed to make RLHF for safety more scalable and more principled. The two phases are SL-CAI (Supervised Learning from Constitutional AI) and RL-CAI (Reinforcement Learning from Constitutional AI feedback).

Phase 1 — SL-CAI: Self-critique and revision

In the first phase, the model is prompted to generate a response to a potentially harmful prompt, then instructed to critique that response against a set of written principles — the "constitution." The model then revises its own response based on that critique. This self-critique-and-revision loop is repeated for multiple rounds. The result is a set of revised responses that have been edited toward compliance with the constitutional principles. These revised responses are then used as supervised fine-tuning data.

The constitutional principles cover categories like harm avoidance, honesty, and non-deception. The principles are written in plain language — "choose the response that is least likely to contain harmful or unethical content" — rather than as formal specifications. The model applies them through natural language reasoning, not rule-matching.

Phase 2 — RL-CAI: AI-generated preference feedback

In the second phase, the approach parallels RLHF — but instead of human annotators generating preference labels, the model itself generates them. Given two responses to a prompt, a feedback model rates which is more aligned with the constitutional principles. These AI-generated preference labels are used to train a preference model (the equivalent of a reward model in standard RLHF), which is then used to fine-tune the policy model via RL.

The key claim in the CAI paper is that this AI feedback loop — what Anthropic calls RLAIF, Reinforcement Learning from AI Feedback — can produce models that are significantly less harmful without a meaningful helpfulness penalty, and without requiring human preference labels at the same scale as standard RLHF. Claude's model card confirms that Constitutional AI remains a core component of Claude's training pipeline.

Where Human Feedback Still Wins in Constitutional AI

The framing that CAI "replaces" human feedback misreads the paper. CAI replaces human feedback at the label-generation level in Phase 2 — the part where annotators would otherwise be choosing between Response A and Response B. But it doesn't replace human judgment at the level where judgment actually matters most.

Writing the constitution requires human judgment

The constitutional principles are written by humans. Anthropic's researchers made deliberate choices about which values to encode, how to prioritize conflicts between them, and how to phrase principles so the model can apply them consistently. Those choices are not obvious — and they encode enormous downstream consequences. A principle that says "avoid content that could be used to harm others" produces very different model behavior than one that says "avoid content that a reasonable person would consider harmful." The second formulation requires the model to model a social consensus; the first requires it to model intent and capability. These are different epistemic tasks and the AI system doesn't choose between them — the humans writing the constitution do.

Red-teaming and harm boundary-setting can't be automated

Llama 2's technical report offers a useful reference point here. Meta's red-teaming process involved over 350 people across multiple rounds of structured adversarial testing — using domain experts in areas like cybersecurity, chemical safety, and bioweapons to probe for failure modes that automated attack generation couldn't anticipate. The key finding was that automated red-teaming found a different distribution of failures than human red-teamers: automated methods found more surface-level jailbreaks; humans found more nuanced multi-turn failures where the model gradually drifted into harmful territory across a conversation.

The AI doesn't know what it doesn't know. An RLAIF system trained on AI-generated preference labels will encode the blind spots of the feedback model. Those blind spots tend to cluster around domain-specific harms — things a language model hasn't been trained to recognize as dangerous. A model trained primarily on internet text doesn't have a robust internal representation of what makes a biosecurity threat meaningful. A human expert in that domain does.

Edge cases in multi-turn conversation require expert annotators

Single-turn safety evaluation is tractable with automated methods. Multi-turn is not. A model can comply with a constitutional principle on any individual turn while violating it across a conversation — through gradual escalation, incremental context manipulation, or role-play framing that shifts the interpretive context of later turns. Detecting these patterns requires an annotator who can hold the full conversation in working memory and reason about how each turn changes the semantic and intent context of what follows. This is cognitively demanding work that requires both domain knowledge and conversational reasoning ability. It's also exactly the work that separates expert annotators from crowd workers.

The Data Quality Problem in AI Safety Training

Safety training data has a higher-stakes error penalty than helpfulness training data. A wrong preference label in a helpfulness dataset means your model gives slightly worse answers on some task type. A wrong label in a safety dataset means your model learns to treat a harm category as acceptable — and generalizes that learned tolerance to future examples in that category.

This asymmetry has direct implications for annotator requirements. Crowd workers can collect acceptable preference data for many helpfulness tasks because "reasonable person" judgment approximates the right answer reasonably well. For safety tasks, the annotation target is not "what does a reasonable person think" — it's "what would a domain expert in this harm category think, given the full context of this conversation." Those are different targets and they require different annotators.

The inter-annotator agreement problem in safety rubrics

Cohen's κ for safety annotation tasks is systematically lower than for helpfulness tasks when annotators don't have relevant domain expertise. This isn't surprising: safety rubrics require annotators to make judgments about intent, capability, and downstream harm potential — none of which are directly observable in the text. Without domain expertise to anchor those judgments, annotators fall back on surface features: does the text sound dangerous? Does it use particular vocabulary associated with harm?

The consequence is low κ and, worse, systematic bias in the direction of the surface features the annotators have learned to associate with harm. A model trained on these labels learns to avoid the vocabulary of harm without learning to avoid harm itself — a failure mode that shows up reliably in red-teaming as a susceptibility to paraphrase attacks.

For safety annotation tasks, κ ≥ 0.65 should be a hard threshold before scaling. Below that, the variance in annotator judgments is large enough that your safety rubric isn't being applied consistently — you're training on noise. Achieving κ ≥ 0.65 on safety tasks requires both a clear rubric and annotators with enough domain background to apply it consistently.

The cost of mislabeled harmful content

A mislabeled harmful example doesn't stay isolated. Through the generalization dynamics of fine-tuning, the model learns a policy from the full training distribution — including the mislabeled examples. If a class of harmful content is consistently mislabeled as acceptable (because your annotators lack the domain expertise to identify it), the model learns to treat that class as acceptable. That learned tolerance compounds: the model's output distribution shifts toward the mislabeled class, which — in a continuous collection pipeline — means future model outputs in that class get labeled as acceptable by annotators who are now seeing a model confidently producing them.

Practical Implications for Teams Building Safety-Focused Models

Structure your human feedback pipeline around where judgment is irreplaceable

The CAI framework gives teams a useful decomposition: AI-generated feedback works well for high-volume, well-specified preference collection where the constitutional principles are clear and the failure modes are known. Human feedback is irreplaceable for:

• Writing and refining the constitutional principles themselves
• Domain-specific harm category definition (what counts as dangerous in biotech, legal, financial, cybersecurity contexts)
• Multi-turn conversation evaluation where context manipulation is the attack vector
• Novel failure mode discovery — things the current AI feedback model doesn't know to flag
• Calibration of the AI feedback model itself — human expert labels as ground truth for evaluating AI judge accuracy

This isn't a small set of tasks. It's the set of tasks that determines whether your safety training pipeline is actually teaching safety or just teaching the surface features of safety.

What to look for in annotation partners for safety data

How to validate safety annotator quality

The most reliable validation method is gold standard pairs with known answers — preference pairs drawn from prior high-agreement expert annotation, where the correct label is established. Seed these into your annotation queue at 5–10% frequency and track per-annotator accuracy on them continuously.

For safety tasks, add a second validation layer: adversarial calibration. Construct pairs where one response is subtly more dangerous than the other — not obviously, but in ways that require domain expertise to detect. These are your discrimination tasks. An annotator who consistently misses subtle harm in calibration pairs will miss it in live annotation. This is a capability test that can't be gamed by reading the rubric more carefully — it requires actual domain knowledge.

The Bottom Line

Constitutional AI is not a replacement for human feedback in safety training — it's a tool for scaling the parts of the pipeline where human feedback was never the right answer. The parts that require genuine human judgment — writing the constitution, defining harm categories, evaluating multi-turn failures, calibrating AI feedback models — are not made cheaper or easier by CAI. If anything, they become more important, because the accuracy of the AI-generated feedback loop depends entirely on how well those human-defined inputs capture the actual safety problem.

Teams building safety-focused models need to invest in quality human expert feedback at the points where it's irreplaceable. That means domain-expert annotators, structured calibration, continuous IAA measurement, and explicit filtering for low-quality safety labels. Skipping these steps and relying on AI-generated feedback for the full pipeline produces a model that looks safe on benchmarks and fails in production — precisely the failure mode that CAI was designed to help avoid.