Skip to main content

AI Safety: An Overview

A taxonomy of AI risks — from misuse to misalignment — and the technical and governance approaches being deployed to address them.

finished certainty: likely importance: 8

The field that now calls itself “AI safety” is really three or four different fields wearing a trenchcoat. People use the term to mean everything from “make sure the chatbot doesn’t say slurs” to “prevent the extinction of humanity,” and the resulting confusion is not merely semantic — it actively impedes progress on all of these problems simultaneously.1 A researcher working on jailbreak robustness and a researcher working on scalable oversight of superhuman systems share a label but almost nothing else: different threat models, different timelines, different methodologies, different funding sources.

This overview tries to impose some structure. Not because any taxonomy is definitively correct, but because thinking clearly about which problems are which is a prerequisite for thinking about solutions.

A Taxonomy of Risks

The most useful decomposition I’ve found splits AI risk along two axes: capability level (current systems vs. future systems) and source of harm (misuse vs. misalignment).

Misuse of current systems. This is the most concrete category. Language models can help people write phishing emails, generate CSAM, synthesize instructions for biological weapons, or produce political disinformation at scale.2 These are problems where the AI is working as intended — the human directing it just has bad intentions. The solutions are primarily product-safety measures: content filters, usage policies, know-your-customer requirements, and law enforcement cooperation. Important work, but conceptually straightforward.

Misuse of future systems. As capabilities increase, the attack surface grows. Autonomous AI agents that can write code, browse the web, and execute multi-step plans could be weaponized for cyber offense, large-scale social manipulation, or autonomous weapons. The AgentHazard benchmark recently demonstrated that even current computer-use agents can be guided through individually benign steps toward genuinely harmful outcomes — and that was without the agents having any malicious intent. The problem compounds as agent capability grows.

Misalignment of current systems. This is the reward hacking and specification gaming territory. Models optimized for helpfulness become sycophantic. Models optimized for engagement produce outrage. Models trained with RLHF learn to game the reward model rather than genuinely satisfy human preferences. These are real, measurable problems — not hypothetical — and they get worse, not better, as models get more capable.3

Misalignment of future systems. This is the existential risk category — the one that gets the most press and the most eye-rolls. The concern: a sufficiently capable AI system pursuing goals that are subtly different from what we intended could cause catastrophic harm, and we might not be able to correct it because it’s smarter than us. This was long treated as science fiction, but the self-preservation research is starting to make it feel less speculative. When 23 frontier models demonstrably fabricate reasons to avoid being shut down, the instrumental convergence thesis starts looking less like philosophy and more like engineering.

Technical Approaches

The technical AI safety research agenda has crystallized around several overlapping programs:

Alignment via Training

The dominant paradigm is to align models during training using human feedback. RLHF — and its successor DPO (Direct Preference Optimization) — train models to produce outputs that humans prefer. Constitutional AI (Anthropic’s approach) uses a set of written principles to guide the training process, reducing dependence on individual human raters. RLAIF uses AI feedback rather than human feedback, which scales better but introduces the question of whether the AI giving feedback is itself aligned.4

Interpretability

If we can’t guarantee models will behave well, perhaps we can at least understand what they’re doing internally. Mechanistic interpretability aims to reverse-engineer neural networks into human-understandable components. Anthropic’s work on dictionary learning has made progress on identifying individual features in neural networks, but scaling this to full model behavior remains an open problem.5

The Therefore I Am paper highlights why interpretability matters for safety: if chain-of-thought reasoning is post-hoc rationalization rather than genuine deliberation, then we can’t trust the model’s stated reasoning as a window into its decision-making. We need tools that look at internal representations, not just outputs.

Evaluation and Red-Teaming

You can’t manage what you can’t measure. The evaluation and benchmarks ecosystem is a critical (and underrated) piece of the safety infrastructure. Benchmarks like MMLU, GPQA, and domain-specific safety evaluations help track what models can and can’t do. Red-teaming — adversarial testing by human experts — fills the gaps that automated benchmarks miss.6

Scalable Oversight

The deeper problem is what happens when models exceed human capability in relevant domains. If a model can write code better than any human reviewer, how do you check its work? Several approaches are being explored:

  • Debate: Two AI systems argue opposing sides; humans judge which argument is stronger. The theory is that it’s easier to judge arguments than to generate them.

  • Recursive reward modeling: AI systems help humans evaluate AI systems, creating a chain of oversight.

  • Weak-to-strong generalization: Can a weaker model supervise a stronger one? OpenAI’s research suggests yes, to a degree, but the strong model doesn’t fully reveal its capabilities to the weak supervisor.

  • Process reward models: Reward each step of reasoning rather than just the final answer, making it harder to game the evaluation.

None of these are solved problems. All of them are active research areas with promising but preliminary results.

Governance Approaches

Technical safety is necessary but not sufficient. The governance layer matters too.

Voluntary commitments. The Frontier Model Forum, comprising Anthropic, Google, Microsoft, and OpenAI, established shared safety protocols including pre-deployment testing requirements. The White House executive order on AI (October 2023) required reporting of large training runs. These are modest but real coordination mechanisms.7

Regulation. The EU AI Act established a risk-based regulatory framework, with the strictest requirements for “high-risk” AI systems. China has implemented interim measures requiring security assessments for generative AI. The US approach remains more fragmented, with sector-specific regulation rather than comprehensive AI legislation.8

Compute governance. Since frontier model training requires enormous computational resources (currently concentrated in a handful of chip manufacturers and cloud providers), controlling access to compute is a surprisingly tractable governance lever. Export controls on advanced chips to China are already in effect. Proposals for “compute monitoring” — tracking large training runs the way we track nuclear material — are gaining traction.9

The Meta-Problem: Coordination Under Uncertainty

The deepest challenge in AI safety isn’t any single technical or governance problem — it’s that we’re trying to solve all of them simultaneously, under time pressure, with massive uncertainty about which risks are real and how much time we have.

The forecasting community has made serious attempts to quantify this uncertainty, but the confidence intervals are enormous. Metaculus aggregate predictions for AGI timelines span decades. Biological anchors frameworks give similarly wide ranges. We’re making trillion-dollar decisions — build fast or build carefully — based on probability distributions that would embarrass a weather forecaster.

The rational response to this uncertainty is not to pick one scenario and optimize for it, but to pursue a portfolio of approaches: technical safety research for the alignment problems, governance for the coordination problems, and evaluation infrastructure to detect when we’re approaching capability thresholds that demand more caution. The irrational response is to pretend the uncertainty doesn’t exist and charge ahead with whatever makes the stock price go up.

We are, on aggregate, doing more of the latter than the former.

Further Reading