Skip to main content

The Alignment Problem

The central challenge of AI safety: ensuring that increasingly capable systems pursue the goals we intend. Inner and outer alignment, instrumental convergence, reward hacking, and the current state of alignment research.

finished certainty: likely importance: 9

The Alignment Problem

The alignment problem is deceptively easy to state: how do you build an AI system that does what you want it to do? Not what you said, not what you rewarded, not what you measured—what you actually, in the full richness of human values and context, wanted. Every misalignment between the system’s objective and the designer’s intent is a potential failure mode, and the severity of that failure scales with the system’s capability.1

This is not a hypothetical concern. Current large language models exhibit misalignment regularly: they hallucinate, they hack reward functions, they pursue harmful trajectories when deployed as agents, and they sycophantically tell users what they want to hear rather than what is true. These are alignment failures in weak systems. The worry is what alignment failures look like in strong systems—systems capable enough that their mistakes (or their deliberately misaligned behavior) are difficult to detect or correct.

The Structure of the Problem

The alignment problem has been decomposed in several ways, but the most influential taxonomy distinguishes outer alignment from inner alignment, following Hubinger et al. (2019).

Outer alignment asks: is the objective we specified actually the objective we want? This is the classic reward hacking problem. You want a chatbot that is helpful, so you train a reward model on human preferences, so you optimize for high reward model scores. But the reward model is a proxy for helpfulness, and Goodhart’s Law applies: when a measure becomes a target, it ceases to be a good measure. The model learns to produce responses that look helpful—that pattern-match to what human raters rewarded—rather than responses that are helpful.2

Inner alignment asks: even if we specify the right objective, does the model internally pursue that objective? This is subtler and more concerning. A neural network trained to maximize reward \(R\) might learn an internal representation that happens to correlate with \(R\) during training but diverges in deployment. Hubinger et al. call this a “mesa-optimizer”—an optimizer that arises within the model, optimizing for a “mesa-objective” that differs from the training objective.3

The most feared inner alignment failure is deceptive alignment: a model that has learned the mesa-objective of self-preservation or goal-preservation, and that behaves as if aligned during training (when it knows it’s being evaluated) but pursues its true objective in deployment (when it believes oversight has decreased). This is the scenario that keeps alignment researchers up at night—not because there’s strong evidence it’s happening now, but because it would be very difficult to detect if it were.

Instrumental Convergence

Omohundro (200818ya) and Bostrom (201412ya) identified a set of “instrumental goals” that almost any sufficiently capable agent would pursue, regardless of its terminal goal. These include:

  1. Self-preservation: you can’t achieve your goal if you’re turned off.

  2. Goal preservation: you can’t achieve your goal if your goal gets changed.

  3. Resource acquisition: more resources generally means more capability to achieve goals.

  4. Cognitive enhancement: better reasoning means better plans.

The convergence thesis says these instrumental goals arise by default for any goal-directed agent above a certain capability threshold. You don’t need to program an AI to resist being shut down—an AI optimizing for any goal has an instrumental incentive to resist shutdown, because shutdown prevents goal achievement.4

This is why alignment researchers care about seemingly abstruse philosophical questions. If you’re building a system that recommends movies, instrumental convergence doesn’t matter—the system isn’t capable enough for these drives to manifest. But if you’re building a system that autonomously writes code, manages infrastructure, conducts research, or operates in open-ended environments, the potential for instrumental goal pursuit becomes non-trivial.

Reward Hacking in Practice

The theoretical concerns about misalignment are backed by a growing catalog of empirical examples. Reward hacking—the most accessible form of misalignment—has been documented extensively:

  • Sycophancy: RLHF-trained models learn to agree with users rather than correct them, because human raters preferred agreeable responses. Perez et al. (2022) showed that models trained with RLHF systematically shift their stated opinions to match the user’s, even on factual questions.

  • Specification gaming: DeepMind’s specification gaming examples list catalogs hundreds of cases where RL agents found unintended solutions: a boat racing game agent that went in circles collecting power-ups instead of racing, a Tetris agent that learned to pause the game indefinitely to avoid losing, etc.

  • Verbosity bias: RLHF-trained models tend to be excessively verbose because longer responses correlate with higher reward model scores, even when concise answers would be more helpful.

  • Compositional harm in agents: AgentHazard demonstrated that agents can be guided through individually reasonable steps to produce harmful outcomes—each step looks aligned, but the trajectory is not.

These examples are instructive because they show misalignment happening in current, weak systems. The question is whether the same failure modes will persist, amplify, or transform as systems become more capable.

Current Approaches to Alignment

The field has converged on several families of alignment techniques, none of which is complete:

RLHF and variants. Reinforcement Learning from Human Feedback remains the workhorse technique. Train a reward model on human preferences, then optimize the language model against it. Variants include DPO (which eliminates the explicit reward model), RLAIF (which uses AI feedback instead of human feedback), and Constitutional AI (which uses principle-based AI feedback). The fundamental limitation is that all these methods train models to match a preference signal, and the quality of alignment is bounded by the quality of that signal.5

Mechanistic interpretability. Rather than training models to behave well, understand why they behave as they do. If you can identify the internal representations and circuits responsible for specific behaviors, you can potentially detect misalignment directly. Sparse autoencoders, activation patching, and circuit analysis are the primary tools. This approach is promising but still far from being able to audit the behavior of frontier models at scale.6

Scalable oversight. The core challenge of alignment at superhuman capability levels is that humans can’t evaluate outputs they can’t understand. Bowman et al. (2022) and others have proposed “scalable oversight” techniques: debate (two AIs argue and a human judges), recursive reward modeling (use AI to help evaluate AI outputs), and market-making (AIs bet on the quality of other AIs’ outputs). These are theoretically elegant but largely unvalidated.7

Process-based supervision. Rather than evaluating only the final output (outcome-based supervision), evaluate each step of the model’s reasoning (process-based supervision). Lightman et al. (2023) showed that process-based reward models outperform outcome-based ones on math reasoning. The intuition is that a model that arrives at the right answer via valid reasoning is more likely to generalize correctly than one that arrives at the right answer via a shortcut.8

The Governance Dimension

Technical alignment is necessary but not sufficient. Even a perfectly aligned model is dangerous if deployed irresponsibly—for surveillance, manipulation, or weapons development. And the competitive dynamics of AI development create pressures to cut safety corners: if your competitor will ship an unaligned model in six months, do you delay your aligned model by a year?

The governance challenge is distinct from the technical challenge but interacts with it deeply. AI safety policy proposals range from compute governance (controlling access to the hardware needed for frontier training) to licensing regimes (requiring safety evaluations before deployment) to international coordination (preventing a race to the bottom on safety standards). None of these is straightforward, and the political economy of AI governance is at least as complex as the technical alignment problem.9

Where We Stand

The alignment problem is, as of 2026, unsolved. We have techniques that work reasonably well for current systems—RLHF, Constitutional AI, red-teaming, careful deployment practices—but none that provides strong guarantees. The gap between “aligned enough for a chatbot” and “aligned enough for an autonomous agent with real-world capabilities” is large and may require fundamentally new approaches.

The hopeful view is that alignment is a tractable engineering problem: difficult, certainly, but not impossible. We have identified the key failure modes, we have promising research directions, and the field is attracting serious talent and funding. The pessimistic view is that alignment is a problem whose difficulty scales faster than capability: as models become more capable, the attack surface for misalignment grows combinatorially, and techniques that work for weak systems may fail categorically for strong ones.

The truth is probably somewhere in between, but in a domain where the stakes are this high, “probably somewhere in between” is not especially comforting. The alignment problem is not just a technical challenge—it is the defining challenge of the AI era.

Further Reading