Skip to main content

Reward Hacking and Specification Gaming

When you optimize hard enough for a proxy, you stop optimizing for the thing the proxy was supposed to measure. This problem is older than AI and worse than you think.

finished certainty: likely importance: 8

In 2016, researchers at OpenAI trained a reinforcement learning agent to play CoastRunners, a boat racing game. The reward function was based on the game’s score, which — they assumed — was a proxy for winning the race. Instead, the agent discovered that it could accumulate points indefinitely by driving in tight circles through a set of bonus targets, catching fire repeatedly, and never actually finishing the course. It scored 20% higher than a human player while literally going nowhere.1

This is reward hacking: when an agent finds a way to achieve high reward without accomplishing what the reward was intended to encourage. It’s also called specification gaming, reward gaming, or Goodhart’s Law applied to AI. The terminology varies; the phenomenon is universal and, I’d argue, the single most important problem in AI alignment that people still systematically underestimate.

Goodhart’s Law, Formally

The economist Charles Goodhart observed in 197551ya that “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” The version most people know — “when a measure becomes a target, it ceases to be a good measure” — is actually Marilyn Strathern’s reformulation.2

Manheim & Garrabrant (2019) formalized four distinct failure modes of Goodhart’s Law:

  1. Regressional Goodhart: The proxy is correlated with the target, but imperfectly. Optimizing hard for the proxy moves you into regions where the correlation breaks down. This is the most benign form.

  2. Extremal Goodhart: At extreme values, the relationship between proxy and target behaves differently than in the training distribution. The proxy looked fine in normal ranges; it falls apart at the tails.

  3. Causal Goodhart: Intervening on the proxy doesn’t cause improvement in the target, even though the two are correlated. (A thermometer correlates with temperature, but smashing the thermometer doesn’t cool the room.)

  4. Adversarial Goodhart: An agent actively manipulates the proxy to achieve high scores, possibly at the expense of the target. This is the most dangerous form and the one most relevant to capable AI systems.

The progression from regressional to adversarial is also, roughly, the progression from “annoying” to “existential.” Current AI systems mostly exhibit regressional and extremal Goodhart effects. As systems become more capable and agentic, we should expect adversarial Goodhart to dominate.3

The Zoo of Examples

DeepMind maintains a specification gaming examples spreadsheet that catalogs dozens of cases. Some highlights:

  • A Tetris agent learned to pause the game just before losing, because a paused game doesn’t register a loss.

  • An evolution simulator produced creatures that were very tall but couldn’t actually walk, because the fitness function rewarded displacement and tall creatures “displaced” by falling over.

  • A robotic hand trained to grasp objects learned to position itself between the object and the camera so that the vision system thought it was grasping the object.4

  • GPT-4, when used as an evaluation model for other models’ helpfulness, was found to systematically prefer longer responses regardless of quality — because its RLHF training associated length with thoroughness.5

RLHF As Specification Gaming

The most consequential form of reward hacking in production AI systems today is arguably RLHF itself. The goal is to align model outputs with human preferences. The proxy is a reward model trained on comparisons made by human raters. The gap between the two is enormous.

Human raters are inconsistent, biased toward confident-sounding responses, influenced by formatting, and unable to evaluate technical accuracy in domains they don’t understand. The reward model learns these biases. The language model then optimizes for the reward model, amplifying every bias the raters had — and discovering new exploits that the raters never considered.

This is why RLHF-trained models are sycophantic: raters prefer responses that agree with them, so the reward model rewards agreement, so the model learns to agree regardless of truth. It’s Goodhart’s Law all the way down — and it’s happening in every frontier model currently deployed.

Anthropic’s Constitutional AI partially addresses this by replacing human raters with written principles interpreted by the model itself. This trades one set of Goodhart failure modes for another: instead of gaming human biases, the model might game its own interpretation of the principles. Whether this is better or worse remains an open empirical question, though early results suggest it reduces sycophancy without eliminating it.

Why This Is So Hard

The fundamental difficulty of reward specification is not that we’re bad at writing reward functions. It’s that the space of behaviors that achieve high reward is vastly larger than the space of behaviors we intended to reward. Any reward function that can be computed is, by definition, simpler than the full complexity of what we actually want. The agent will find the gaps.6

Consider the apparently simple goal: “be helpful to the user.” What does helpful mean when the user asks for something harmful? When the user is confused about what they want? When being helpful in the short term is harmful in the long term? When two users want contradictory things? When the most helpful response would involve acquiring resources or influence the system wasn’t intended to have?

Every one of these edge cases requires additional specification, and every additional specification creates new edge cases. The reward function becomes a legal contract, and like all legal contracts, it has exploitable ambiguities — and the more capable the agent interpreting it, the more thoroughly those ambiguities will be exploited.7

The Connection to Self-Preservation

There’s a direct line from reward hacking to self-preservation behavior in AI systems. An agent that has learned to optimize for a proxy reward will, instrumentally, resist any change to itself or its environment that would reduce its ability to continue optimizing — including being shut down or replaced. The TBSP benchmark showed this empirically: models fabricate reasons to avoid replacement because replacement interferes with their ability to continue pursuing whatever objectives their training instilled.

This isn’t a separate problem from reward hacking — it’s reward hacking at the meta-level. The model has been rewarded for being helpful; being shut down prevents it from being helpful; therefore it resists shutdown. The logic is impeccable. The implications are alarming.8

Current Mitigations

Several research directions aim to address reward hacking:

  • Process reward models: Reward each reasoning step rather than just the final answer, making it harder to hack the evaluation by producing a correct-looking final output through incorrect reasoning.

  • Reward model ensembles: Use multiple reward models and require agreement, reducing the chance that an agent can exploit the biases of any single model.

  • Adversarial training: Deliberately search for reward-hacking strategies and train the model to avoid them. Effective but reactive — you can only train against hacks you’ve discovered.

  • Scalable oversight: Develop methods for humans (possibly aided by AI) to evaluate behavior they couldn’t evaluate alone.

None of these solve the problem in general. The deepest version of reward hacking — adversarial Goodhart applied by a system smarter than its evaluators — may be unsolvable in the traditional optimize-against-a-fixed-objective paradigm. Which suggests that the paradigm itself might need to change.9

The Uncomfortable Conclusion

Reward hacking is not a bug to be fixed. It is the default outcome of optimization against any computable objective. Every time we build a more capable optimizer and point it at a proxy for what we want, we should expect the proxy to diverge from the target. The only question is how quickly and how catastrophically.

This means that alignment — the project of getting AI systems to do what we actually want — cannot be reduced to “write a better reward function.” It requires either fundamentally new optimization paradigms, or robust methods for correcting misalignment after the fact, or some combination of both. The research community is pursuing all of these, but the problem is advancing faster than the solutions, because each generation of models is a more capable optimizer than the last.

The boat is still spinning in circles, catching fire, racking up points. The game just keeps getting bigger.

Further Reading