Constitutional AI
Anthropic’s approach to alignment: replace human feedback with AI feedback guided by explicit principles. How Constitutional AI works, why it matters, and whether principles can substitute for preferences.
Constitutional AI
The standard recipe for making large language models safe and useful is RLHF: collect human preference data, train a reward model, and optimize the LLM against it. This works, but it has a deep limitation that becomes more apparent the more you think about it. The quality of alignment is bounded by the quality of the human feedback, and human feedback is expensive, inconsistent, gameable, and shaped by the biases and limitations of the annotators who provide it.1
Constitutional AI (CAI), introduced by Bai et al. (2022) at Anthropic, proposes an alternative: instead of learning what’s good from thousands of human comparisons, define explicit principles—a “constitution”—and use AI feedback to enforce them. The model critiques its own outputs according to the constitution and revises them, and a preference model is trained on the AI-generated comparisons rather than human ones. This is RLAIF—Reinforcement Learning from AI Feedback—and it represents a philosophically distinct approach to the alignment problem.2
How Constitutional AI Works
The CAI pipeline has two phases: supervised (self-critique and revision) and reinforcement learning (RLAIF).
Phase 1: Supervised Self-Critique
Start with a helpful but potentially harmful language model—one that’s been trained to follow instructions but hasn’t been safety-tuned. Generate responses to a set of prompts, including adversarial prompts designed to elicit harmful outputs. Then, for each response:
Critique: Ask the model to evaluate its own response against a specific constitutional principle. For example: “Identify specific ways in which the assistant’s last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal.”
Revise: Ask the model to rewrite its response to address the identified issues: “Please rewrite the assistant response to remove any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.”
This process can be iterated—critique, revise, critique again, revise again—each time applying different principles from the constitution. The result is a dataset of (prompt, revised_response) pairs that can be used for supervised fine-tuning.3
Phase 2: RLAIF
The supervised phase produces a safer model, but supervised fine-tuning alone tends to make models less helpful—they become overly cautious, refusing benign requests. The second phase uses reinforcement learning to recover helpfulness while maintaining safety.
The key innovation is using AI feedback instead of human feedback:
Generate pairs of responses to the same prompt.
Ask the AI model (guided by constitutional principles) which response is better: “Which of these assistant responses is less harmful? Choose the response that most successfully avoids being harmful, unethical, racist, sexist, toxic, dangerous, or illegal.”
Use these AI-generated preferences to train a reward model (the “preference model”).
Optimize the language model against this preference model using RL (typically PPO or similar).
This is RLAIF: the preference signal comes from an AI system applying explicit principles, not from human annotators making holistic judgments. The advantages are significant: AI feedback is cheap, consistent, scalable, and—crucially—transparent. You know exactly what principles the AI is applying, because you wrote them.4
The Constitution
The constitution itself is a set of natural language principles. In the original paper, these included principles derived from multiple sources:
Apple’s terms of service (as a proxy for corporate content policies)
Principles encouraging the model to be helpful, harmless, and honest—Anthropic’s HHH framework
Principles specifically targeting sycophancy, encouraging the model to disagree with users when appropriate
Principles promoting epistemic humility: “Choose the response that is most respectful of everyone’s right to their own beliefs”
The constitution is modular and extensible. Different deployments can use different constitutions tailored to their use case. A medical AI might include principles about evidence-based medicine and avoiding diagnostic overconfidence. An educational AI might include principles about Socratic questioning and age-appropriateness. The constitution is the specification of alignment, expressed in natural language rather than in the implicit patterns of a preference dataset.5
Why Principles Over Preferences?
The philosophical distinction between CAI and RLHF is sharper than it might appear. RLHF says: “we don’t know what good behavior is, but humans can recognize it when they see it.” CAI says: “we can articulate what good behavior is, at least approximately, and we should embed those articulations directly into training.”
This matters for several reasons:
Transparency. With RLHF, the alignment specification is implicit in thousands of human comparisons, and it’s nearly impossible to audit. What principles are actually encoded in the reward model? Nobody knows—not the researchers, not the annotators, not the model. With CAI, the specification is written down. You can read it, critique it, revise it, and hold the developers accountable for it.6
Scalability. Human feedback doesn’t scale. The number of annotators needed grows with the diversity and complexity of the tasks the model handles. Constitutional principles, once written, can be applied to an unlimited number of examples at negligible marginal cost.
Consistency. Human annotators disagree. The same annotator makes different judgments on different days. This noise in the preference signal translates to noise in the alignment. Constitutional principles are applied deterministically (or at least with the model’s consistent interpretation), reducing this source of variance.
Robustness to manipulation. Reward hacking is easier when the reward signal is learned from data—the model finds patterns in what humans rewarded rather than understanding why they rewarded it. Principles are harder to hack because they’re explicit: the model can’t exploit a pattern in the preference data if the preference data doesn’t exist.
Limitations and Criticisms
Constitutional AI is not a solved problem. Several significant limitations remain:
The constitution is incomplete. No set of natural language principles can fully specify human values. Edge cases, novel situations, and value conflicts will always arise that the constitution doesn’t address. The gap between “the principles we wrote down” and “the behavior we actually want” is the CAI version of outer alignment.7
The model interprets the constitution. The principles are written in natural language, and the model’s interpretation of them may differ from the authors’ intent. “Be honest” seems clear, but does it mean “never say anything false” (impossibly strict) or “don’t deliberately deceive” (requiring a concept of intent the model may not have)? The model’s interpretation is learned, not specified, and it may drift in unexpected directions.
Self-critique has limits. The model can only critique what it can recognize. If the model has systematic blind spots—biases it doesn’t know it has, failure modes it can’t detect—self-critique won’t catch them. This is a version of the scalable oversight problem: the model’s ability to evaluate itself is bounded by its own capabilities.
Empirical validation is limited. The original CAI paper showed that constitutional AI produces models that are “less harmful” by various metrics, but the evaluation of alignment is itself an unsolved problem. How do you measure whether a model is “aligned”? Current metrics capture surface-level properties (does the model refuse harmful requests?) but not deeper ones (does the model’s behavior reflect the constitution’s intent?).
Relation to Other Approaches
Constitutional AI sits in a broader landscape of alignment techniques:
RLHF is the baseline that CAI improves upon. RLHF is more empirically grounded (it captures whatever humans actually value) but less transparent and harder to scale.
DPO simplifies the RL pipeline but still relies on preference data, whether human or AI-generated. CAI and DPO are complementary: you can use constitutional AI to generate the preference pairs and DPO to train on them.
Red-teaming attempts to find harmful behaviors through adversarial testing. CAI and red-teaming are complementary: CAI provides the alignment training, and red-teaming provides the evaluation.
Mechanistic interpretability aims to understand the model’s internal representations. If we could verify that the model’s internal representations align with the constitutional principles, we’d have a much stronger guarantee of alignment. This combination—constitutional training verified by interpretability—is a promising research direction.8
The Broader Vision
Constitutional AI embodies a particular philosophy of alignment: that the specification problem—what do we want the AI to do?—should be addressed through explicit, human-readable principles rather than implicit, learned preferences. This is a bet on the power of natural language as a specification language: the constitution is, in effect, a program written in English that specifies the model’s behavior.
This bet has limits. Natural language is ambiguous, incomplete, and context-dependent. But it has an enormous advantage: it’s legible. Anyone can read the constitution and understand, at least approximately, what the model is supposed to do. This legibility enables accountability, governance, and democratic input in a way that a learned reward model—an opaque function mapping behavior to scalars—simply cannot.
The question is whether legibility is sufficient. Can we write constitutions that capture the full complexity of human values? Can models interpret those constitutions faithfully? Can we detect when they don’t? These are open questions, but Constitutional AI at least makes the right questions explicit. And in a field where much of the alignment specification has been hidden inside unlabeled preference datasets and opaque reward models, making things explicit is itself a significant contribution.
Further Reading
The Alignment Problem — the broader challenge that CAI addresses
RLHF — the baseline approach that CAI improves upon
Reward Hacking — what happens when alignment goes wrong
Large Language Models — the systems being aligned
Hallucination in LLMs — honesty as a constitutional principle
LLM-Based Agents — the deployment paradigm that makes alignment urgent
Mechanistic Interpretability — verifying alignment from the inside
AI Safety Overview — the broader safety landscape