Attention Is All You Need
On the Transformer paper (Vaswani et al 2017): the architecture that ate the world, why dropping recurrence was the unlock, and why nine years later nobody has found anything fundamentally better.
There are papers, and then there are papers. Most research is incremental, a few points on a benchmark, a clever trick that gets cited fifty times and then quietly forgotten. “Attention Is All You Need” ( et al 2017) is not one of those. By any honest accounting it is one of the handful of most consequential machine learning papers ever written. It introduced the Transformer, and in doing so it reorganized the entire field of artificial intelligence around a single, almost offensively simple idea: you do not need recurrence. You do not need convolutions.1 You just need attention.
I want to tell the story of why this paper mattered so much, because the architecture diagrams and the math, important as they are, do not capture the feeling of the moment. In 2017 the dominant paradigm for sequence modeling was the recurrent neural network, specifically its gated variants: LSTMs and GRUs. These were good. They worked. People had spent years engineering them, tuning them, stacking them, bolting attention onto them.2 The whole NLP pipeline, machine translation and language modeling and summarization, ran on recurrent networks. Then eight researchers at Google, in a paper with one of the best titles in the history of science, said: throw all of that away.
What gets me, reading the paper again in 2026, is how little it sounds like the most important document of the decade. It reads like a solid translation paper. It is modest, narrow, almost apologetic about its scope. The authors had no idea they were lighting a fuse. Or if they did, they hid it well.
The Eight Who Walked Out
Start with the people, because the human story here is stranger than the technical one. The paper has eight authors, and as Wikipedia now notes with a kind of deadpan understatement, “after the paper was published by Google, each of the authors left the company.” Every single one. The team that built the engine of the modern AI industry scattered to the wind, mostly into startups that are now worth billions.
Wired ran a long backstory on this a couple of years ago that is worth reading in full. One detail that stuck with me, surfaced by an HN commenter (mlmonkey) quoting the piece, is how much the thing depended on one person actually making it run. Noam Shazeer joined the effort late and rewrote the code from scratch. “These theoretical or intuitive mechanisms, like self-attention, always require very careful implementation, often by a small number of experienced ‘magicians,’ to even show any signs of life,” Jakob Uszkoreit told Wired. Shazeer “took the basic idea and made the thing up myself,” then came back and said, “Look, it works.” There is a lesson in that about the gap between an idea and a result, and how often the gap is one stubborn engineer.
The nationalities of the authors are their own small story. A commenter (_fizz_buzz_) listed them out: Ashish Vaswani (India), Niki Parmar (India), Jakob Uszkoreit (Germany), Llion Jones (Wales), Aidan Gomez (Canada), Łukasz Kaiser (Poland), Illia Polosukhin (Ukraine), Noam Shazeer (USA). Eight people, seven countries, one Google office in Mountain View. The single most valuable idea in modern computing came out of a profoundly international team, which is worth remembering whenever someone tells you talent is a national resource you can wall off.
And then they all left. The most pointed version of the irony came from a commenter (rvnx): “It seems the authors of the Transformer paper almost all left Google. I wouldn’t call that a success. It shows that the tool was right but the corporate environment wasn’t.” Llion Jones held on the longest. Shazeer left to found Character.AI, got reacquired by Google in a 2.7 billion dollar talent deal in 2024, was made a Gemini co-lead, and has since reportedly left again. Polosukhin founded NEAR Protocol. Gomez co-founded Cohere. Google invented the future and then watched it walk out the door, which is a recurring theme in the company’s history and a thing a lot of HN commenters bring up with some bitterness. As one (tyre) put it: “the amount of original and foundational research at Google is a black mark against their position now. They blew a hell of a lead.”
What Was Wrong With Recurrence
To understand why the Transformer was revolutionary, you have to understand the bottleneck it broke. An RNN processes a sequence one token at a time, left to right.3 The hidden state at position \(t\) depends on the hidden state at position \(t-1\). This is elegant. It mirrors how humans read. It also has two consequences that turned out to be fatal.
First, you cannot parallelize it. Training on modern hardware (GPUs, TPUs) is all about doing thousands of operations at once.4 But an RNN forces you to wait: you cannot compute step 50 until you have finished steps 1 through 49. On long sequences this is painfully slow. LSTMs fixed the gradient-flow problem5 but did nothing for the sequential bottleneck. The dependency chain is the architecture. You cannot buy your way out of it with more silicon.
Second, long-range dependencies are hard. Even with gating, information has to crawl through many sequential steps to get from one end of a sentence to the other. By the time the model reaches the end of a long paragraph, the early context has been compressed, smeared, and partly forgotten. The attention mechanism of Bahdanau et al (201412ya) helped a great deal. It let the decoder look back at every encoder state directly instead of through a single fixed-length bottleneck vector. But the encoder was still recurrent. The crawl remained.
How three architectures connect a sequence. A convolutional layer (left) sees a fixed local window; a recurrent layer (middle) threads information through a chain of sequential states; self-attention (right) connects every position to every other position directly. The collapse of that path length from O(n) to O(1) is the whole game. Diagram: Zhang, Lipton, Li & Smola (d2l.ai), CC BY-SA 4.0.
The Transformer’s bet was a question: what if attention is not an add-on to recurrence but a replacement for it? What if you rip out the recurrent spine entirely and let every token talk to every other token in one step? That is the whole idea. Everything else in the paper is plumbing in service of it. Good plumbing, carefully done, but plumbing.
I find the diagram on the right more illuminating than any equation, so it is worth sitting with. A convolution sees a local neighborhood and has to stack many layers to widen its view. A recurrent net threads a single thread of state through the whole sequence, so position 1 reaches position 100 only after 99 hops. Self-attention wires every position to every other position in a single layer. The maximum distance any piece of information has to travel drops from the length of the sequence to a constant. That structural fact, more than any benchmark number, is why the thing won.
The Architecture
The original paper describes an encoder-decoder architecture for machine translation. The encoder ingests a source sentence, the decoder produces the target. Both are built entirely from attention and feedforward layers, with no recurrence anywhere. Let me walk through the pieces, because they are each worth understanding on their own.
The full Transformer as drawn in the paper: encoder stack on the left, decoder stack on the right, each layer a sandwich of attention, feedforward, residual connections and layer normalization. The paper stacks six of each. Diagram: Yuening Jia, CC BY-SA 3.0.
Self-Attention, the Core Mechanism
The heart of the Transformer is self-attention, which the paper calls “Scaled Dot-Product Attention.” For each token you compute three vectors: a query (\(Q\)), a key (\(K\)), and a value (\(V\)), each from a learned linear projection.6 Attention is then:
\[\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V\]
What does this mean? Each token “queries” every other token: how relevant are you to me? The dot product between a query and a key measures relevance. The softmax turns those scores into a probability distribution. Each token’s output is then a weighted average of all the value vectors, weighted by relevance. That is the entire operation. Every token attends to every other token in a single parallel step, no sequential crawl required.
Scaled dot-product attention drawn out: queries and keys produce a relevance matrix, softmax normalizes it, and the result re-weights the values. The whole sequence is processed at once. Diagram: Numiri, CC BY-SA 4.0.
The \(\sqrt{d_k}\) scaling factor7 is a small but load-bearing detail. Without it, dot products grow large in high dimensions and shove the softmax into saturated regions where the gradient is nearly zero. Divide by the square root of the key dimension and the variance stays sane. It is the kind of unglamorous fix that separates a paper that works from one that does not.
Multi-Head Attention
Rather than one attention function, the Transformer runs several in parallel (the paper uses \(h=8\)), each with its own learned projections, then concatenates and projects the results. The intuition is that different heads can specialize: one might track syntactic dependencies, another semantic similarity, another simple positional adjacency. It costs barely more compute than single-head attention because the per-head dimension shrinks proportionally, and it is meaningfully more expressive. Years of interpretability work has since gone into figuring out what individual heads actually learn, with mixed and frequently surprising results.
Positional Encoding
Here is a subtle problem. Self-attention is permutation-equivariant.8 It treats its input as a set, not a sequence. “The cat sat on the mat” and “mat the on sat cat the” would produce identical attention patterns, which is obviously wrong for language. To inject order, the paper adds positional encodings, sinusoids of different frequencies, to the input embeddings. The trick is elegant: because the encoding at position \(p+k\) can be written as a linear function of the encoding at position \(p\), the model can learn to attend by relative position.
This is the part of the paper that one prominent voice thinks is the actual contribution. An “Attention Is All You Need” coauthor said in 2024 he was “sick” of transformers, and a commenter (Mithriil) in that thread argued: “its most important idea is the Positional Encoding. The transformer head itself is just another NN block among many.” I am not sure I fully buy that, but it is a useful corrective to the idea that attention is the only thing in the room. Positional encoding is what lets a set-processing mechanism handle order at all, and a surprising amount of later research (RoPE, ALiBi) is just better answers to the same question.
The Full Stack
Each encoder layer is: multi-head self-attention, then a residual connection and layer normalization, then a feedforward network, then another residual and layer norm. The decoder adds a cross-attention layer (attending to the encoder’s output) and masks future positions in its self-attention so generation stays autoregressive. The paper stacks six of each. Residual connections9 are not optional decoration here; they are the gradient highways that make a deep stack trainable at all.
Deep Dive: Why Attention Scales and Recurrence Does Not
The computational advantage of the Transformer is not only about parallelism. It is about the complexity of the operations themselves. Take a sequence of length \(n\) with hidden dimension \(d\):
RNN: \(O(n)\) sequential operations, each \(O(d^2)\). Total work \(O(n \cdot d^2)\), but the critical path (the longest unavoidable dependency chain) is \(O(n)\). You cannot shorten that path with more hardware. It is baked into the model.
Self-attention: \(O(1)\) sequential operations, everything in parallel, working over an \(n \times n\) attention matrix. Total work \(O(n^2 \cdot d)\), with a critical path of \(O(1)\).
For typical 2017 NLP, \(n\) (sequence length, often 50 to 512) was much smaller than \(d\) (model dimension, 512 to 1024), so self-attention was both faster in wall-clock terms and had shorter dependency paths. The quadratic cost in \(n\) only bites for very long sequences, which is exactly what spawned an entire sub-field of efficient attention: Linformer, Performer, FlashAttention, and dozens more.
The deeper point is about information flow. In an RNN, for token 1 to influence token 100, information traverses 99 nonlinear transformations and degrades along the way. In a Transformer, token 1 attends to token 100 directly, in one layer. The maximum path length between any two positions is \(O(1)\) per layer. The Transformer’s superiority at long-range dependencies is not a lucky empirical finding. It is a structural guarantee you can read straight off the architecture.
The Results, And Why They Were Understated
The paper evaluated on machine translation: the WMT 2014 English-to-German and English-to-French benchmarks. The Transformer hit state-of-the-art BLEU scores10 (28.4 on EN-DE, 41.0 on EN-FR) while training far faster than the competition. The big model trained in 3.5 days on eight P100 GPUs. Prior state-of-the-art systems had taken weeks.
Here is what is funny in retrospect. The paper was modest. It pitched the Transformer as a better machine-translation architecture. The authors clearly suspected it was more general (the title is “Attention Is All You Need,” not “A Better Translation Model”), and per Wikipedia they did experiment with using it to generate Wikipedia articles and to do parsing, which convinced them it was a general-purpose language model. But the published validation was narrow. Translation BLEU and a parsing table. Nobody, I think, fully anticipated what came next, and that includes the authors. A commenter (dinfinity) made the point sharply: the paper “didn’t ever use the term LLM or large language model because the phrase didn’t exist in industry.” They were not writing the founding document of the LLM era. They thought they were writing a translation paper. The founding document is what it became.
There is a real historical irony worth naming, because it cuts against the romance of the lone-genius narrative. The Transformer was not conjured from nothing. As one commenter (asdff) noted, the paper has 32 references, “like most ~10 page manuscripts.” Bahdanau-style attention was four years old. Residual connections, layer norm, feedforward blocks, byte-pair encoding: all existed. Another (stared) put it well, listing the Transformer next to AlexNet as papers that “are refinements of existing ideas, and show how they help.” The genius was not inventing a new primitive. It was knowing which primitives to keep, which to throw out, and having the nerve to throw out the one everyone assumed was load-bearing.
What Happened Next
What happened next was everything.
Within about a year, two lines of work split off and reshaped the field. GPT ( et al 2018) took the decoder half, trained it as an autoregressive language model on a big text corpus, and showed the representations could be fine-tuned for downstream tasks. That was the birth of the pretrain-then-finetune paradigm. BERT ( et al 2018) took the encoder half, trained it with masked language modeling (predict the randomly hidden tokens), and flattened benchmarks across the board. BERT showed that bidirectional11 pretraining produced representations good enough that a simple classifier on top could beat hand-built task-specific architectures.
Then GPT-2 (2019) showed that scaling the decoder produced eerily fluent text. Then GPT-3 (2020) showed that at sufficient scale the model could do tasks from a prompt alone, no fine-tuning, what we now call in-context learning. Then the Vision Transformer (2020) showed it worked on images. Then AlphaFold 2 (2021) used attention to crack protein structure prediction. Then DALL-E, Stable Diffusion, Whisper, ChatGPT, GPT-4, Claude, Gemini, Llama, and the DeepSeek models.
Every one of these is a Transformer, or a close cousin. The architecture did not just win machine translation. It won everything: language, vision, audio, protein structure, robotics, code. As of 2026 there is essentially no competitive deep learning system in any of those domains that is not a Transformer or a lightly modified descendant. The paper has been cited more than 250,000 times, which puts it among the ten most-cited papers of the 21st century. For a seven-page architecture paper, that is absurd.
Deep Dive: The Scaling Hypothesis, or Why the Architecture Mattered Less Than Everyone Thought
One of the strangest lessons of the post-Transformer era is what gets called the scaling hypothesis: performance is mostly a function of compute, data, and parameters, and the architectural details matter less than you would expect as long as the base architecture is good enough to scale.
The Transformer turned out to be “good enough to scale” in a way nothing before it was. RNNs could not scale because of the sequential bottleneck. Convolutional models parallelized fine but had limited receptive fields. The Transformer’s mix of global attention and full parallelizability meant you could keep throwing compute at it and keep getting returns, returns that Kaplan et al (2020) later formalized as neural scaling laws.
This is a slightly unsettling finding, because it suggests the key contribution of “Attention Is All You Need” was not any one clever trick. It was removing the bottleneck that had been preventing scale. The authors built the thing for translation tasks with a few hundred tokens of context. They were not designing for GPT-4-scale runs on trillions of tokens. But the properties that made it good at translation (parallelism, direct long-range connections, stable training under residuals and layer norm) happened to be exactly the properties you need to train hundred-billion-parameter models. Sometimes the most important thing a paper does is not the thing its authors set out to do.
You can hear the cynical, almost deflationary version of this all over practitioner forums. A commenter (qudat) put it bluntly: LLMs “are all designed essentially the same, with minor differences in how they are trained. They are all feed-forward multi-headed attention models; it doesn’t matter if it’s a 4B model or a 1T model, that’s just scale.” Another (sysreq_) noted that frontier models “are still just tuning 2017’s ‘Attention is All You Need’ with MoE, RLHF, and more compute.” There is real truth in this, and also a trap. Yes, the core block has barely changed. But “just scale” is doing a tremendous amount of work in those sentences, and the engineering required to actually scale (data pipelines, KV-cache tricks like DeepSeek’s MLA, quantization, distributed training) is its own mountain of invention.
The Paper Is Not a Good Way to Learn the Thing It Started
Here is a contrarian take I have come around to. If you want to understand modern LLMs, “Attention Is All You Need” is a surprisingly bad place to start, and saying so out loud is a small act of heresy.
The clearest statement of this came from a commenter (sigmoid10) reacting to someone trying to learn LLMs from the original paper: it “is actually a bad paper if you want to learn about autoregressive LLMs specifically, because it describes a more complicated encoder-decoder architecture while modern LLMs are decoder only. So it’s an unnecessarily hard way to get into the subject.” They recommended the GPT-2 paper instead, “the first time LLMs really showed what is possible,” and pointed out that there was “more to it than just ‘scaling the transformer algorithm’ like many people claim.”
This is correct and worth internalizing. The thing the world runs on now is the decoder-only Transformer: take the right half of the original diagram, drop the cross-attention, train it to predict the next token, scale. The full encoder-decoder design in the paper is translation-shaped, with machinery that most modern LLMs simply do not have. People come to the 2017 paper expecting the blueprint for ChatGPT and instead find a translation system with two interlocking stacks. The blueprint for ChatGPT is GPT-2 and GPT-3, which are in some sense simplifications of the original. Even Simon Willison argues that diving into “transformer models, positional encodings and self-attention at the very start of an introduction to LLMs” does little to help a beginner, and that “they can predict the next word” gets you further faster.
There is something fitting about a foundational paper being a poor introduction to its own legacy. The Transformer was not the destination. It was the thing that made the destination reachable. Reading it to understand GPT-5 is a bit like reading the patent for the internal combustion engine to understand a Formula 1 car. The lineage is real and the core is recognizable, but an enormous amount happened in between, and most of what makes the modern thing work is not in the founding document.
Has Anything Beaten It?
Nine years on, the honest answer is: not really, and people keep trying.
The most credible challenger is the state space model line, above all Mamba (Gu & Dao, 2023), which scales linearly with sequence length instead of quadratically and has generated genuine excitement.12 There was a real wave of it on HN in early 2024, with Jamba as a production-grade Mamba-Transformer hybrid. The pitch is seductive: keep the parallel-training story, drop the quadratic attention cost, win on very long context. And yet. The dominant production systems are still Transformers, or hybrids that keep attention layers in the mix. The pattern so far is not “Mamba replaces the Transformer” but “Mamba layers get interleaved with attention layers because attention is still doing something the SSM cannot.” The barren middle ground, where you can feel the field wanting a successor and not quite finding a clean one, is itself a testament to how good the original bet was.
What strikes me, reading nearly a decade of these discussions, is the mix of awe and exhaustion. One coauthor is publicly “sick” of transformers. Practitioners are tired of the architecture being treated as the end of history. And still, every quarter, the best model in the world is a Transformer with a slightly different coat of paint: rotary embeddings, RMSNorm, SwiGLU, grouped-query attention, FlashAttention, mixture-of-experts routing. The core, the thing in the 2017 paper, is still in there, recognizable, doing the heavy lifting.13
What I Think About It
I think “Attention Is All You Need” is that rare paper that changes the topology of a field’s possibility space. Before the Transformer, the landscape of sequence modeling was rugged and boxed in. You could optimize within the RNN paradigm, but you could not escape its fundamental limits. The Transformer did not just find a higher peak in the existing terrain. It revealed that the terrain was much larger than anyone realized, with whole continents reachable only through parallelizable architectures.
The paper also embodies something I keep relearning about research: the best ideas are often the least complicated ones. Every component (attention, residuals, layer norm, feedforward blocks) existed before 2017. The contribution was combining them correctly and deleting the unnecessary parts.14 The authors did not add complexity. They removed it. They threw away recurrence, the thing everyone assumed you could not do without, and what was left was faster, simpler, and better. That is a much harder and braver kind of work than piling on machinery, and it is consistently undervalued because it does not look like effort.
I do not want to over-romanticize it. The cynics on HN have a point: the core trick was a refinement of prior art, the real magic was scale and a few stubborn engineers, and the paper is a lousy LLM tutorial. All true. But step back far enough and the basic fact is hard to dodge. Eight people, seven countries, one office, one seven-page paper with a cocky title, and the entire trajectory of artificial intelligence bent around it. They built it for translation. It ate the world.
As of 2026 we are still living inside the world this paper made. Every major AI system is a Transformer. Nine years of the smartest people in the field hammering at it, and nobody has found anything fundamentally better. That is the signature of a paper that got something deeply, structurally right. Attention, it turns out, really was most of what you needed.
The original paper: Attention Is All You Need by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Published at NeurIPS 2017. The human backstory is told in Steven Levy’s Wired feature, “Eight Google Employees Invented Modern AI.” Related reading on this site: BERT, GPT-3, scaling laws, the Transformer architecture explainer, recurrent networks and LSTMs, and quantum computing for a very different take on a hyped frontier.