The Scaling Hypothesis
The idea that ate AI: bigger models, more data, more compute, predictably better performance. From Kaplan to Chinchilla to GPT-4 and beyond. Where the scaling hypothesis works, where it breaks, and what it means for the future.
The Scaling Hypothesis
The scaling hypothesis is the claim that the primary driver of progress in deep learning, particularly in language models, is scale: more parameters, more training data, more compute. Not better architectures. Not cleverer training tricks. Not domain-specific inductive biases. Just more. Increase the inputs to the system along well-understood axes, and capabilities will improve smoothly and predictably according to power laws.
This is, on its face, a deeply unsatisfying idea. It says the bottleneck to artificial intelligence is not insight but resources. That the path to better AI runs not through conceptual breakthroughs but through bigger GPU clusters and larger datasets. That the Transformer architecture from 2017, essentially unchanged, is sufficient, and what was missing was just the will and the money to make it very, very large.
Why power laws look like straight lines on log-log axes: a square-root (concave) curve becomes a line of slope 0.5, a square (convex) curve a line of slope 2. Plotting loss against compute the same way is what turned “bigger is better” from folklore into something you can extrapolate. Illustrative figure.
And yet, from roughly 2019 to 2025, the scaling hypothesis was the single most successful predictive framework in AI research. It correctly predicted that GPT-3 would be qualitatively better than GPT-2. It correctly predicted that training on more data would help more than making models bigger, which is the Chinchilla result. It correctly predicted that capabilities would keep improving with scale even when individual benchmarks saturated. It is the closest thing the field has to a theory, and it has shaped decisions involving tens of billions of dollars of compute investment.
Whether it will continue to hold is the most important open question in AI. I want to walk through where the idea came from, where the evidence is strongest, where it might break, and why some of the people who built it now think the easy gains are over.
Where the Idea Came From
The scaling hypothesis did not arrive as a single insight. It accreted out of a decade of people being repeatedly embarrassed by the same mistake: betting on cleverness and losing to brute force.
The intellectual root is Rich Sutton’s bitter lesson, written in 2019, which looked back at seventy years of AI research and found a depressing regularity. Researchers would build in human knowledge, hand-crafted features, clever priors, domain-specific structure, and it would help for a while. Then someone would throw more compute at a more general method and win. It happened in computer chess, where brute-force search beat elegant positional heuristics. It happened in computer Go with AlphaGo. It happened in speech recognition and computer vision once deep learning and big datasets showed up. Sutton’s conclusion stung: the methods that win are the ones that scale with computation, and our intuitions about how to inject intelligence by hand are mostly a trap.
The empirical hints came earlier than the equations. Hestness et al. (2017) at Baidu measured how error fell as datasets and models grew across translation, language modeling, and image classification, and found power-law scaling that held across domains. The paper got far less attention than it deserved at the time. It was, in retrospect, the scaling laws result a few years before the field was ready to believe it.
What made people believe was GPT-2. In early 2019, OpenAI trained a 1.5-billion-parameter language model that wrote coherent paragraphs, and the reaction was a mix of awe and alarm, including a famously staged release over fears of misuse. The architecture was not novel. It was a scaled-up Transformer. The lesson a lot of people took away was that the model got dramatically better mostly because it got bigger. Gwern wrote the canonical long-form articulation of this view, arguing that the GPT line was a demonstration that intelligence-like behavior could be a smooth function of scale, and that the field was systematically underestimating how far the curve would go. That essay shaped how a generation of researchers thought about the problem, and it is still the best statement of the strong version of the hypothesis.
Then the equations arrived, and the folklore became a research program.
The Empirical Foundation: Kaplan Et Al. (2020)
The scaling hypothesis existed as folklore before it had equations. People noticed that bigger models worked better. GPT-2 beat GPT-1, and BERT-Large beat BERT-Base. But “bigger is better” is a vague claim. How much bigger? How much better? Where should you spend your next dollar?
Kaplan et al. (2020), the scaling laws paper, turned the vague claim into precise equations. They showed that cross-entropy loss on language modeling follows power laws as a function of model parameters \(N\), dataset size \(D\), and compute budget \(C\):
\[L(N) \propto N^{-0.076}, \quad L(D) \propto D^{-0.095}, \quad L(C) \propto C^{-0.050}\]
These exponents are small. You need roughly 10× more compute to halve the loss. But they are remarkably stable across several orders of magnitude. Plot loss against any of these variables on log-log axes and you get a straight line. That regularity is what turns an empirical observation into something you can plan around.1
The scaling hypothesis, in its strong form, says these power laws will keep holding. There is no wall. There is no qualitative change in the nature of what the model can do, just smooth, predictable improvement. Capabilities that seem to “emerge” at certain scales are not discontinuities in the underlying loss curve but artifacts of how we measure performance on discrete tasks.2
The Chinchilla Correction
Kaplan et al. got the scaling laws right but the allocation wrong. Their analysis suggested that when you get more compute, you should mostly make the model bigger: \(N \propto C^{0.73}\), \(D \propto C^{0.27}\). This advice directly shaped GPT-3 (175B parameters, 300B tokens) and the arms race toward ever-larger models that followed: Gopher (280B), Megatron-Turing NLG (530B), PaLM (540B).
Hoffmann et al. (2022), the Chinchilla paper, showed that parameters and data should scale equally: \(N \propto C^{0.50}\), \(D \propto C^{0.50}\). Under this corrected allocation, GPT-3 was massively undertrained. A 175B model should have seen about 3.5 trillion tokens, not 300 billion. Conversely, 300B tokens would be optimally served by a 70B model. Chinchilla, at 70B parameters trained on 1.4T tokens, matched or beat the 280B Gopher on nearly every benchmark while being four times smaller.3
The discrepancy came from a subtle methodological issue in Kaplan et al.’s setup. They used a fixed learning rate schedule across different training lengths, which systematically disadvantaged longer training runs. See the scaling laws paper review for a detailed treatment.
The Chinchilla correction matters because it shifted the field’s strategy from “make the model as big as possible” to “train the model on as much data as possible.” LLaMA (Meta, 2023) trained a 65B model on 1.4T tokens, far beyond Chinchilla-optimal, because inference cost matters more than training cost for a widely-deployed model. LLaMA 3 went further: 70B parameters on 15T tokens. The trend toward smaller, longer-trained models is a direct consequence of taking scaling laws seriously and asking “optimal for whom?” The answer is optimal for the company paying the electricity bill at inference time, not just at training time.4
Where Scaling Works
The evidence for the scaling hypothesis is strongest in language modeling, but it extends well beyond:
Vision. Scaling laws for image classification (Zhai et al., 2022) showed power-law improvement for Vision Transformers. Diffusion models scale predictably too. The jump from DDPM to Stable Diffusion to SDXL to Flux tracks closely with increases in compute and data.
Code. Codex (2021) and subsequent code models show consistent improvement with scale. AlphaCode 2 at competitive-programming level, Cursor and Copilot in production, all powered by scaled-up language models.
Multimodal. GPT-4V, Gemini, and Claude demonstrate that scaling works for models that process both text and images. The pattern generalizes: the Transformer architecture, scaled up with enough multimodal data, learns to handle multiple modalities without modality-specific engineering.
Reasoning. Perhaps the most striking evidence comes from reasoning tasks. Chain-of-thought prompting (Wei et al., 2022) only works reliably above a certain model scale. Mathematical reasoning, as measured by benchmarks like GSM8K and MATH, improves steadily with scale. Models that can barely do arithmetic at 1B parameters can solve competition-level problems at 100B+.
The Emergence Debate
One of the most contested questions in scaling research is whether emergent capabilities (abilities that appear suddenly at certain scales) are real phenomena or measurement artifacts.
Wei et al. (2022) documented numerous examples of emergence: tasks where performance is near-random for small models and then jumps abruptly above a critical model size. These included multi-step arithmetic, word unscrambling, and various forms of abstract reasoning. The implication was dramatic: scaling doesn’t just make models quantitatively better, it produces qualitatively new capabilities.
Schaeffer et al. (2023) challenged this narrative. They argued that emergence is a mirage created by nonlinear or discontinuous evaluation metrics. If you measure accuracy (0 or 1 per problem), a smooth improvement in per-token probabilities can look like a sudden jump when the model crosses the threshold of getting the whole problem right. Switch to a smoother metric (like per-token log-likelihood) and the emergence disappears, and performance improves smoothly and predictably at all scales.
My read: Schaeffer et al. are mostly right for the cases they analyze. The underlying loss curve is smooth. But I think there are genuine cases where capabilities compose in ways that produce real qualitative transitions. When a model becomes good enough at multiple sub-skills simultaneously, it can suddenly perform tasks that require all of them. This is less emergence in the physics sense and more a manifestation of the conjunction of independent improvements, but the practical effect, new capabilities appearing at scale, is real.
The implication for safety is obvious. If capabilities can appear unpredictably at certain scales, then scaling up a model might produce dangerous capabilities that didn’t exist at smaller scales. If emergence is smooth but our metrics are coarse, we might still be surprised, not by the model but by our own failure to measure it properly. Either way, the lesson is: scale cautiously, and measure carefully.
Where Scaling Breaks (Or Might Break)
The scaling hypothesis is not a law of nature. It’s an empirical regularity observed over a particular range of scales, with particular architectures, on particular tasks. There are several reasons it might fail:
Data walls. Scaling laws require data to scale alongside compute. But high-quality text data is finite. Estimates suggest there are roughly 10–20 trillion tokens of high-quality English text on the public internet.5 Current frontier models are already trained on a significant fraction of this. Muennighoff et al. (2023) showed that repeating data degrades performance, since you can’t just loop through the internet twice and get the same benefit. Epoch AI’s analysis suggests we may exhaust high-quality language data by 2028.
The responses to this are synthetic data (train on model-generated text, which has its own problems), multimodal data (images, video, audio contain vast untapped information), and data quality over quantity (Gadre et al., 2024 showed that data quality scaling laws differ substantially from quantity laws).
Diminishing returns. The power-law exponents are small. \(L \propto C^{-0.05}\) means 10× more compute buys you a 12% reduction in loss. At some point, the return on additional compute investment may fall below the threshold of economic viability. We’re not there yet, since each increment of scale still produces commercially valuable improvements, but the curves are bending.6
Reasoning ceilings. Language models trained with next-token prediction may have fundamental limitations on the kind of reasoning they can perform. Bachmann & Nagarajan (2024) argue that autoregressive models cannot perform certain types of compositional reasoning regardless of scale. The counterargument is that chain-of-thought, tool use, and test-time compute scaling (letting models “think longer” at inference) may circumvent these limitations. As of 2026, the evidence is mixed. Models are still getting better at reasoning with scale, but the rate of improvement on hard reasoning benchmarks may be slowing.7
Architecture limitations. The Transformer’s quadratic attention cost limits context length. The lack of persistent memory means models can’t learn from experience during deployment. These are architectural constraints that no amount of scaling fixes. Whether they matter for AGI-relevant capabilities is debatable, but they’re real limitations of the current paradigm.
Test-Time Compute: A New Axis of Scaling
A development that complicates the simple scaling picture is test-time compute scaling: the idea that you can improve model performance not by making the model bigger or training it longer, but by giving it more compute at inference time.
OpenAI’s o1 (2024) demonstrated this dramatically. By training models to produce long chains of reasoning and search over multiple solution paths, o1 achieved large improvements on math and coding benchmarks that couldn’t be reached just by scaling the base model. Essentially, the model is doing a form of beam search over reasoning chains, trading inference compute for accuracy.
This opens a genuinely new axis of scaling. Instead of the traditional \(C_{\text{train}} \approx 6ND\) relationship, you now have \(C_{\text{total}} = C_{\text{train}} + C_{\text{inference}} \times \text{num\_queries}\), and optimizing over all three axes simultaneously changes the optimal allocation. Snell et al. (2024) showed that for certain tasks, it’s more efficient to use a smaller model with extensive test-time compute than a larger model with a single forward pass.
This is philosophically interesting because it breaks the identification of “intelligence” with “model size.” A small model that thinks carefully can outperform a large model that answers immediately. The scaling hypothesis in its crude form (bigger model = better) becomes wrong. The refined version (more total compute = better) may still hold, but the optimal allocation shifts from training to inference as models get deployed on harder tasks.
The Argument on the Ground
Benchmark scores climbing across model generations. Each jump tracks roughly with an order-of-magnitude increase in training compute, which is the visual signature the scaling hypothesis predicts. Illustrative composite figure.
I find the practitioner discussion more honest than the papers, because the people arguing have money or reputations on the line. The scaling hypothesis was never settled in a seminar room. It was settled, to whatever extent it has been, by people watching their loss curves and their cloud bills.
The skeptics have a real case, and it is worth taking seriously rather than dismissing. A recurring worry is convergence. As one commenter, ldjkfkdsjnv, put it on Hacker News: “all of the models (opus, llama3, gpt4, gemini pro) are converging to similar levels of performance. If it was true that the scaling hypothesis was true, we would see a greater divergence of model performance.” This is the strongest version of the bear case. If scale were the whole story, the lab with the most compute should be pulling away, and instead the frontier looks like a tight pack. The counterargument is that the leaders are all roughly Chinchilla-trained on roughly the same internet, so they converge because their inputs converge, not because scale stopped mattering. But I don’t think the skeptic is wrong to find the clustering suspicious.
The believers, meanwhile, keep pointing at the parts nobody saw coming. Another commenter, ivape, noted that “no one predicted the emergent properties of LLMs as they followed the scaling laws hypothesis,” and that we “don’t actually know what would emerge if we trained a quadrillion param model.” That uncertainty cuts both ways. It is the reason scaling enthusiasts stay optimistic and the reason safety researchers stay nervous. Nobody has a principled cap on what falls out of the next order of magnitude.
There is also a more textured middle position, which is roughly where I land. A commenter going by aspenmartin described progress as “a mix, performance gains are bursty but we have been getting a lot of bursts (RLVR, test-time compute, agentic breakthroughs).” That matches what I see. The smooth loss curve is real, but the capability curve people actually care about arrives in lumps, because capabilities are thresholded conjunctions of skills. The plumbing improves smoothly. The thing on top of the plumbing improves in jumps.
What strikes me reading these threads is how little the disagreement is about the math. Almost nobody disputes that loss follows a power law. The fights are about extrapolation, about whether the line keeps going, and about whether loss reduction translates into the capabilities that matter. Those are not questions the Kaplan paper can answer. They are bets, and people are placing them with very different amounts of conviction.
The Philosophical Stakes
The scaling hypothesis is not merely a technical claim. It carries deep implications for how we think about intelligence, about who gets to build AI, and about the future.
If intelligence, or something functionally equivalent, is primarily a matter of scale, then:
There are no conceptual breakthroughs needed for AGI. The path is “just” engineering: more compute, more data, better training infrastructure. This is the position associated with Ilya Sutskever’s famous claim that “scale is all you need,” and it has been remarkably predictive over the 2020–52025 period.
AI development concentrates in the hands of a few. If the primary input is compute, and compute costs hundreds of millions to billions of dollars, then frontier AI is built by a handful of organizations: OpenAI, Google DeepMind, Anthropic, Meta AI, and perhaps a few others. Academic labs cannot compete on the main axis of progress. This has already happened. The gap between frontier models and the best academic models is measured in orders of magnitude of compute.8
Safety becomes urgent. If capabilities improve predictably with scale, and scale is increasing rapidly, then the timeline to human-level (or superhuman) performance on various tasks can be estimated. These estimates are uncertain, but they exist, and many of them point to uncomfortably near futures. The alignment problem, ensuring that scaled-up models do what we want, becomes not a philosophical thought experiment but an engineering deadline.
Rich Sutton’s bitter lesson is the philosophical ancestor of the scaling hypothesis: “The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective.” The scaling hypothesis is the bitter lesson with numbers attached. The numbers have held up. The lesson remains bitter.
What I Think About It
I think the scaling hypothesis is mostly right for the current paradigm (Transformer-based models trained on internet text) and importantly incomplete as a general theory of intelligence.
It is mostly right because the empirical evidence is overwhelming. From GPT-2 to GPT-4, from Stable Diffusion 1.5 to Flux, from early RLHF to Constitutional AI, the models that scale more have consistently performed better. Every prediction of a “wall,” that language models would plateau, that image generation would saturate, that reasoning was beyond reach, has been wrong so far. Betting against scale has been a losing strategy for six consecutive years.
It is importantly incomplete because I believe there are aspects of intelligence (grounding, persistent memory, causal reasoning, genuine novelty) that may require more than just making the current paradigm bigger. The scaling hypothesis says nothing about what scales, only that scaling works within a given paradigm. If the Transformer + next-token prediction paradigm has fundamental limitations, then the scaling curves will eventually flatten, and the next S-curve will require new ideas.
But I notice that I’ve been wrong before. Every time I’ve thought “surely scale can’t solve this,” scale has solved it. The correct epistemic stance may be: assume scaling works until proven otherwise, but keep looking for the proof.
There is one more thing I keep turning over, which is what happens to research culture under a regime like this. If the dominant lever is compute, then the most valuable researchers are not the ones with the best ideas but the ones with access to the biggest clusters. That is a strange and slightly demoralizing thing to say out loud, and it runs against the romantic story the field tells about itself. The bitter lesson is bitter precisely because it devalues the kind of insight that academics are trained to produce. I think a lot of the resistance to the scaling hypothesis is not really about the evidence. It is about not wanting the evidence to be true, because if it is true then cleverness matters less than capital. I share that discomfort. It does not make the curves bend.
The most important thing the scaling hypothesis has done is give the field legibility. Before scaling laws, AI progress was unpredictable. After scaling laws, it’s on a curve. You can extrapolate. You can plan. You can estimate when certain capabilities will emerge and prepare for them. This predictability is, for safety researchers, both a gift (we can see what’s coming) and a curse (what’s coming may arrive before we’re ready). Either way, ignoring the scaling hypothesis in 2026 is no longer a credible intellectual position. You can argue about where it breaks. You can’t argue that it hasn’t worked.
Key references: Kaplan et al. (2020), the original scaling laws. Hoffmann et al. (2022), the Chinchilla correction. Gwern’s “The Scaling Hypothesis”, the best long-form treatment. Epoch AI compute trends, ongoing empirical tracking.