Skip to main content

Large Language Models: A Survey

From GPT-1 to frontier models: the trajectory of large language models, the scaling laws that predicted their rise, the emergent abilities that surprised everyone, and the RLHF techniques that made them useful. A survey of where we are and what remains unsolved.

finished certainty: likely importance: 9

Large Language Models: A Survey

The story of large language models is, at bottom, a story about a bet. The bet was that a single architecture—the Transformer—trained on enough text with enough parameters would produce something qualitatively different from what came before. Not just better autocomplete, but a system that could reason, translate, code, and converse. As of 2026, that bet has paid off spectacularly, though the returns are distributed unevenly and the bills are still coming due.1

This essay surveys the trajectory: from the GPT lineage through scaling laws, emergent abilities, alignment via RLHF, and the current state of the field. It is not exhaustive—the literature is now measured in tens of thousands of papers—but it traces the load-bearing ideas.

The GPT Lineage

The lineage begins with GPT-1 (2018), which demonstrated that unsupervised pretraining on a large text corpus followed by supervised fine-tuning could achieve strong performance on diverse NLP benchmarks. The model was 117M parameters—tiny by current standards—but the conceptual move was enormous: rather than designing task-specific architectures, you train a general-purpose language model and adapt it.2

GPT-2 (2019) scaled to 1.5B parameters and introduced the “zero-shot” paradigm: the model could perform tasks it was never explicitly trained on, simply by conditioning on natural language descriptions. OpenAI famously delayed releasing the full model due to concerns about misuse—a decision that was widely mocked at the time and looks increasingly prescient in retrospect.

GPT-3 (2020) was the inflection point. At 175B parameters, trained on 300B tokens, it demonstrated in-context learning: the ability to perform new tasks from just a few examples provided in the prompt, without any gradient updates. This was genuinely surprising. Nothing in the training objective—next-token prediction—explicitly optimized for this capability. It appeared to emerge from scale.3

The subsequent generations—GPT-4 (2023), GPT-4 Turbo, GPT-4o—continued improving, though OpenAI stopped publishing architectural details. The trend is clear: each generation is more capable across a wider range of tasks, but the rate of improvement per dollar of compute may be slowing.

Meanwhile, the open-source ecosystem exploded. Meta’s LLaMA (2023) demonstrated that smaller models trained on more data could match much larger ones—a direct consequence of the Chinchilla insight that most early LLMs were severely undertrained. Mistral, Qwen, DeepSeek, and others followed, and by 2025 open-weight models were competitive with proprietary ones on many benchmarks.

Scaling Laws and the Scaling Hypothesis

The theoretical backbone of the LLM revolution is the scaling hypothesis: the claim that model performance improves predictably as a power law of compute, parameters, and data. Kaplan et al. (2020) formalized this, and Hoffmann et al. (2022) corrected the optimal allocation (the “Chinchilla” correction), showing that parameters and data should scale equally.4

The power of scaling laws is that they are predictive. You can estimate the loss of a 100B-parameter model from experiments on 1B-parameter models. This transforms AI research from empirical tinkering into something closer to engineering: you can plan investments, estimate returns, and make resource allocation decisions with reasonable confidence.

But scaling laws predict loss, not capabilities. A model’s cross-entropy loss on a held-out text corpus improves smoothly with scale. Its performance on specific tasks—coding, math, reasoning—may improve smoothly, or it may appear to jump discontinuously. This distinction is at the heart of the debate over emergent abilities.

Emergent Abilities

Wei et al. (2022) documented what they called “emergent abilities”—capabilities that appear to arise suddenly at certain model scales. Below a threshold, the model performs at chance on a task; above it, performance jumps to well above chance. Examples include multi-step arithmetic, chain-of-thought reasoning, and word-level manipulation tasks.

This finding generated enormous excitement and concern. If capabilities can appear unpredictably at certain scales, then we cannot reliably forecast what the next generation of models will be able to do. This has obvious implications for AI safety and for the alignment problem.5

The resolution may be that emergence is real at the task level but illusory at the capability level. A model might gradually improve at arithmetic (a capability) but appear to suddenly acquire the ability to add 5-digit numbers (a task, measured by exact match). The underlying function is smooth; the measurement is discontinuous.

RLHF and the Alignment Turn

Raw language models are not useful products. They ramble, lie, produce toxic content, and follow instructions inconsistently. The bridge from “impressive demo” to “useful tool” was Reinforcement Learning from Human Feedback (RLHF), formalized by Ouyang et al. (2022) in the InstructGPT paper.

The RLHF pipeline has three stages: (1) supervised fine-tuning on high-quality demonstrations, (2) training a reward model on human preference comparisons, and (3) optimizing the language model against the reward model using PPO or similar algorithms. This procedure dramatically improved instruction-following, reduced harmful outputs, and made models more “helpful, harmless, and honest”—Anthropic’s formulation that has become an industry standard.6

RLHF’s success spawned variants. Direct Preference Optimization (DPO) simplified the pipeline by eliminating the explicit reward model. Constitutional AI replaced human feedback with AI feedback guided by explicit principles. RLAIF generalized the idea further. Each approach trades off different failure modes, but the core insight remains: pretraining gives you capabilities; alignment training gives you a usable product.

Architecture and Training Innovations

The Transformer remains the dominant architecture, but it has been refined substantially. Key innovations include:

  • Rotary Position Embeddings (RoPE): Su et al. (2021) introduced a position encoding that enables better length generalization, now standard in most open models.

  • Mixture of Experts (MoE): Sparsely-activated models like Mixtral and DeepSeek-V2 use conditional computation to scale parameter counts without proportionally scaling inference cost.

  • Grouped Query Attention (GQA): Reduces the memory overhead of multi-head attention during inference, enabling longer context windows. Used in LLaMA 2 and subsequent models.

  • Flash Attention: Dao et al. (2022) restructured the attention computation to be IO-aware, dramatically reducing memory usage and enabling longer sequences.

Context windows have expanded from 2048 tokens (GPT-3) to 128K+ tokens in current models, with some experimental architectures handling millions of tokens. This is not merely a quantitative change—it enables qualitatively different applications like full-codebase understanding and long-document analysis.7

Multimodal Extensions

Modern LLMs increasingly handle multiple modalities. GPT-4V and Claude 3 accept images as input. Gemini was designed from the ground up as a multimodal model. Audio, video, and structured data are being integrated into the same framework.

The trend is toward unified models that handle all modalities through a single architecture, rather than bolting vision encoders onto language models. Whether this convergence is architecturally inevitable or merely convenient remains an open question.

Current State and Open Problems

As of early 2026, the field is characterized by several tensions:

Scaling vs. efficiency. The scaling hypothesis continues to hold in aggregate, but the cost curve is brutal. Training frontier models costs hundreds of millions of dollars. Inference at scale costs billions per year. The push toward smaller, more efficient models—via distillation, MoE, quantization, and better data curation—is as important as the push toward scale.8

Capabilities vs. reliability. LLMs can do astonishing things but cannot be relied upon for any specific thing. Hallucination remains pervasive. Factual accuracy is inconsistent. Reasoning breaks down on novel problems. The gap between “can do X in a demo” and “reliably does X in production” remains wide.

Open vs. closed. The open-weight ecosystem (LLaMA, Qwen, Gemma, DeepSeek) is increasingly competitive with proprietary models. This has implications for safety (open models can’t be recalled), for innovation (open models enable more research), and for market structure (commoditization pressures margins).

Chatbots vs. agents. The first wave of LLM deployment was conversational—chatbots that answer questions. The second wave is agentic—systems that take actions, use tools, and operate autonomously over extended periods. LLM-based agents raise fundamentally different challenges around safety, reliability, and alignment. The transition from chatbot to agent is arguably the most important shift happening in the field right now.9

What Comes Next

Prediction is hazardous, but a few trends seem robust. Models will continue to get more capable, but the marginal return per FLOP may decrease. Multimodality will become the default. Agents will become the primary deployment modality. And the alignment problem—how to ensure these systems do what we want, especially as they become more autonomous—will only become more pressing.10

The LLM revolution is, in a sense, still in its early chapters. The architecture is mature, the scaling laws are well-understood, and the basic training recipe is established. What remains is everything else: reliability, safety, efficiency, and the much harder problem of building systems that are not just capable but trustworthy. The survey of LLMs is, increasingly, a survey of the problems they create.

Further Reading