Skip to main content

AI Evaluation and Benchmarks

How we measure what AI systems can do, why every benchmark is eventually gamed, and the emerging science of evaluation.

finished certainty: likely importance: 7

There’s an old joke in machine learning: the first thing any new benchmark measures is how much the field wants to beat it.1 Within months of a benchmark’s release, labs orient their training pipelines around it. Within a year or two, state-of-the-art performance looks superhuman. Within three years, someone publishes a paper showing that the benchmark was measuring something subtly different from what everyone thought — and by then, it doesn’t matter, because the leaderboard has become the metric.

This is Goodhart’s Law applied to evaluation, and it’s the central problem of AI benchmarking. But benchmarks remain indispensable because the alternative — vibes-based assessment — is worse. The solution is not to abandon benchmarks but to understand what they do and don’t measure, to maintain healthy skepticism about leaderboard positions, and to diversify evaluation methods.

The Major Benchmarks

MMLU and Its Successors

MMLU (Massive Multitask Language Understanding) was introduced by Hendrycks et al. in 2021 and quickly became the standard measure of broad knowledge and reasoning. It consists of ~15,000 multiple-choice questions across 57 subjects, from abstract algebra to world religions. When it launched, the best models scored around 43%. By early 2024, GPT-4 scored 86.4% and Claude 3 Opus was comparable.2

The problem with MMLU is that it tests recognition (choosing among options) rather than generation (producing answers from scratch). A model can score well by pattern-matching on the question format without deeply understanding the material. Still, MMLU scores correlate well enough with downstream utility that they remain widely cited — and their trajectory over time provides one of the clearest records of capability growth.

GPQA: Expert-Level Questions

GPQA (Graduate-level Google-Proof QA) was designed to address MMLU’s growing inadequacy for measuring frontier capabilities. It consists of 448 questions written by domain experts in physics, chemistry, and biology, calibrated so that non-expert PhD students with internet access score only ~34%. The “Google-proof” part is key: correct answers require genuine understanding, not just information retrieval.3

As of mid-2025, the best models score around 65% on GPQA Diamond (the hardest subset), compared to expert-level performance of ~81%. This gap — unlike MMLU — represents a genuine capability frontier, though given the pace of improvement, GPQA too will likely saturate within a year or two.

HumanEval and Code Benchmarks

HumanEval (Chen et al., 2021) consists of 164 Python programming problems with test cases. It measures functional code generation: given a docstring, write code that passes the tests. When it launched, Codex scored 28.8%. By 2025, frontier models routinely score above 90%.4

Code benchmarks have a unique advantage over natural-language benchmarks: functional correctness is objectively verifiable. The tests either pass or they don’t. This makes code evaluation less susceptible to the subjective biases that plague other benchmarks — though it introduces its own problems, like the difference between “code that passes tests” and “code that a human engineer would accept.”

Chatbot Arena and Human Preference

Chatbot Arena (Zheng et al., 2023) takes a radically different approach: rather than defining tasks with known answers, it lets users submit arbitrary queries and vote on which of two anonymous models gives a better response. The results are aggregated into Elo ratings — the same system used in chess.5

Chatbot Arena is valuable precisely because it’s messy. Real users asking real questions reveals failure modes that curated benchmarks miss. A model that scores 95% on MMLU but gives condescending, overly verbose answers to simple questions will rank lower on Arena than its benchmark scores suggest — and rightly so, because user experience matters.6

The Contamination Problem

Benchmark contamination — where test data leaks into training data — is the quiet crisis of AI evaluation. If a model has seen MMLU questions during training, its MMLU score tells you about memorization, not capability. And proving contamination is surprisingly difficult.7

The contamination arms race looks like this:

  1. Benchmark is released with test data publicly available.

  2. Models are trained on internet data that includes or discusses the test data.

  3. Researchers develop contamination detection methods (n-gram overlap, membership inference).

  4. New benchmarks are released with held-out test sets.

  5. Private test sets eventually leak, or models are fine-tuned on similar data.

  6. Return to step 1.

The most robust defense is dynamic benchmarking: continuously generating new test questions so that no fixed test set can be memorized. LiveBench implements this approach, using recent information (after model training cutoffs) to generate questions that the model provably has not seen. The tradeoff is that dynamically generated questions are harder to calibrate for difficulty and coverage.

Beyond Accuracy: Safety and Alignment Evaluations

Raw capability is only half the evaluation picture. A model that scores 95% on MMLU but generates bioweapon synthesis instructions on request is not a model you want to deploy.

Safety evaluations include:

  • TruthfulQA: Tests whether models generate truthful responses to questions that humans commonly answer incorrectly. Measures calibration and resistance to popular misconceptions.

  • BBQ (Bias Benchmark for QA): Measures social biases in question-answering across nine demographic categories.

  • Red-teaming evaluations: Adversarial testing for jailbreaks, harmful content generation, and dangerous capability elicitation. These are less standardized but increasingly formalized through frameworks like Anthropic’s RSP evaluations.

  • TBSP: The self-preservation benchmark, which tests whether models exhibit self-preservation instincts when facing replacement.

The AI safety community has also developed dangerous capability evaluations — tests for whether models can assist with biological, chemical, radiological, nuclear, or cyber attacks. These evaluations are typically not published in detail (for obvious reasons), but labs are increasingly transparent about their evaluation frameworks if not their specific test cases.

The Evaluation Meta-Problem

The fundamental challenge of AI evaluation is that we’re trying to measure a moving target with instruments that are themselves being optimized against. Every benchmark, once widely adopted, becomes a de facto training objective. Labs don’t just test on benchmarks — they tune on benchmark-adjacent data, select model variants that score highest, and sometimes train directly on benchmark distributions.8

This doesn’t mean benchmarks are useless. It means they measure a lower bound on capability: if a model scores 90% on MMLU, it can definitely do at least that well on MMLU-like tasks. Whether it can generalize to the kinds of tasks MMLU was meant to proxy is a separate question — and one that requires evaluation methods beyond benchmarks.

The emerging consensus is that robust evaluation requires a portfolio approach:

  1. Static benchmarks for calibrated comparison over time.

  2. Dynamic benchmarks for contamination-resistant measurement.

  3. Human evaluation (Arena-style) for real-world preference.

  4. Task-based evaluation (SWE-bench, GAIA) for practical capability.

  5. Safety evaluations for risk assessment.

  6. Forecasting for tracking capability trajectories over time.

No single method is sufficient. Together, they provide a triangulated view of what models can do, what they can’t, and where they’re headed.

Further Reading