The Transformer Architecture

Mert

The Transformer Architecture

The architecture that ate the world. How self-attention, positional encoding, and the encoder-decoder structure replaced recurrence, why it really won (hardware, not elegance), and what it still gets wrong.

2026-04-06–2026-06-21 finished certainty: likely importance: 10

The Problem the Transformer Was Actually Solving
The Architecture
Why It Actually Won (And It Isn’t Elegance)
What the Transformer Gets Wrong
The Variants Zoo
- Encoder-Only, Decoder-Only, and Encoder-Decoder
The Challenger: Is Anything Going to Replace It?
The Monoculture Worry
Further Reading

If you want to understand modern AI, not at the level of cocktail-party patter but actually understand what is happening inside GPT-4, Claude, Gemini, and every other large language model, you need to understand exactly one architecture. Not recurrent networks. Not convolutional networks. The Transformer.¹

I’ve written separately about the original paper that introduced it, Vaswani et al’s “Attention Is All You Need” (2017). That piece is about the paper’s context and what it set off. This one is the how it works article, the explainer I wish I’d had when I first sat down with the diagram and felt my eyes slide off it. I’ll assume you know roughly what a neural network is and have some vague sense of backpropagation, and nothing past that.

A thing worth sitting with before the math: this architecture was written by eight people at Google, and at the time nobody upstairs thought it was a big deal. As one of the authors, Jakob Uszkoreit, told Wired later, the higher echelons of Google “saw the work as just another interesting AI project,” and the team mostly got left alone. The paper has since been cited more than 250,000 times, which puts it among the ten most-cited papers of the entire 21st century. Every one of the eight authors has since left Google. The thing they built quietly rearranged the industry they worked in. I find that gap between “just another project” and “rewired civilization” genuinely hard to hold in my head, and I think it should make you humble about anyone’s ability to call which paper matters in real time.²

The Problem the Transformer Was Actually Solving

Here is the first thing that surprised me when I went back to the sources: the Transformer was not designed to build a general intelligence. It was designed to translate German.

To see what it did, you have to see what it replaced. Recurrent neural networks process a sequence one token at a time, left to right. At each step \(t\), the hidden state \(h_t\) depends on the previous hidden state \(h_{t-1}\) and the current input \(x_t\). In theory this is lovely. The hidden state carries forward a compressed memory of everything seen so far. In practice it was a disaster for three reasons.

Sequential computation kills parallelism. You cannot compute \(h_{100}\) until you have computed \(h_1\) through \(h_{99}\). That means an RNN cannot exploit the thing that actually makes modern deep learning fast: massively parallel GPU hardware. This turns out to be the whole ballgame, and I’ll keep coming back to it.

Long-range dependencies decay. Even with gating tricks like LSTMs and GRUs, information from early tokens gets diluted as it passes through hundreds of sequential steps. The vanishing gradient problem makes it genuinely hard for the network to learn that token 5 matters for predicting token 500.

The hidden state is a bottleneck. Everything about the past has to be squeezed into one fixed-size vector. It’s like summarizing a novel in a single sentence and then being quizzed on the footnotes.³

The attention mechanism showed up first as a patch for that third problem, not as a replacement for anything. Bahdanau et al (2014_12ya) showed that a translation decoder, instead of leaning on a single final hidden state, could “attend” to all of the encoder’s hidden states at each step and pull out whatever was relevant. It helped enormously. But it was still bolted onto an RNN spine, so the sequential bottleneck stayed.

This is a point the practitioner community is careful about in a way the popular coverage usually isn’t. As the Hacker News commenter littlestymaar put it, “Attention heads existed before Transformers, they were used in recurrent neural networks (RNN) to improve their performance. The paper is called ‘Attention is all you need’ because transformers keep the attention head while discarding the RNN part entirely.” Another commenter, PartiallyTyped, drew the line even more precisely: “prior to transformers it was RNN + Attention (+ Residuals), but the paper claimed that Attention (+Residuals+Pos Embeddings) is all you need.” The radical move was subtractive. Throw the recurrence away. Keep attention. See what happens.

The seq2seq predecessor: an RNN encoder-decoder with an attention mechanism bolted on. The decoder (bottom) attends back over the encoder’s hidden states (top) at each step. The Transformer kept the attention part and deleted the recurrent chains. Animation: Google, Apache License 2.0, via Wikimedia Commons.

The Architecture

The original Transformer follows an encoder-decoder structure. The encoder reads the whole input sequence and produces a sequence of continuous representations. The decoder takes those and generates the output one token at a time.⁴

The full Transformer, both stacks. Left: the encoder. Right: the decoder, with its extra cross-attention block reaching back into the encoder’s output. Every box labeled “Multi-Head Attention” is the same operation pointed at different inputs. Diagram: dvgodoy, CC BY 4.0, via Wikimedia Commons.

Both the encoder and the decoder are stacks of identical layers. In the original paper, \(N = 6\) of each. The layers within a stack share structure but not weights, and the encoder and decoder layers differ in one component I’ll get to. If the canonical diagram on the right looks like too much at first, you’re in good company. The HN commenter minihat said it plainly: “I have not seen an all-in-one diagram and cannot imagine one being helpful, since there’s too much going on.” So let’s not start from the diagram. Let’s build up to it from the one operation that matters.

Self-Attention: the Core Operation

The beating heart of the Transformer is scaled dot-product self-attention. The idea: for every token in a sequence, work out how much it should attend to every other token, then replace that token’s representation with a weighted blend of all the others.

Concretely, each token’s embedding is projected into three vectors: a query (\(Q\)), a key (\(K\)), and a value (\(V\)). The attention score between two tokens is the dot product of one’s query with the other’s key. A high dot product means these two tokens are relevant to each other.⁵

The full formula:

\[\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V\]

The \(\sqrt{d_k}\) scaling in the denominator stops the dot products from growing too large, which would shove the softmax into a region where its gradients are almost zero and training stalls.⁶

The payoff is the whole point of the architecture: every token can directly attend to every other token in a single step. Information doesn’t have to crawl through a chain of hidden states. Token 1 and token 500 are equally close, with a path length of \(O(1)\) instead of the RNN’s \(O(n)\). That is the structural fix for the long-range-dependency problem, and it falls out of the design for free rather than being patched in.

What one attention head actually computes: queries and keys produce a matrix of pairwise scores, softmax turns them into weights, and those weights blend the values into each token’s new representation. Diagram: Shuang Zhang et al, CC BY 4.0, via Wikimedia Commons.

Multi-Head Attention

A single attention function can only express one kind of relationship at a time. Multi-head attention runs \(h\) attention operations in parallel (the paper uses \(h = 8\)), each with its own learned projections, then concatenates the results and mixes them with one more linear layer:

\[\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, ..., \text{head}_h) W^O\]

where each head is \(\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)\).

This lets the model attend to several aspects of the input at once: one head might track syntax, another semantic similarity, another sheer positional proximity. The lovely thing is that mechanistic interpretability research has since confirmed heads really do specialize, often in ways a human can read.⁷

Positional Encoding

Self-attention has a strange property: it is permutation-invariant. On its own it has no idea that token 3 comes before token 4. Without something extra, “dog bites man” and “man bites dog” produce identical representations, which is obviously fatal for language. The Transformer fixes this by adding sinusoids of different frequencies to the input embeddings:

\[PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d_{model}})\] \[PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i/d_{model}})\]

The sinusoidal choice was clever: because of how sines and cosines combine, the model can learn to reason about relative positions through linear transformations. Later work explored learned positional embeddings, RoPE, and ALiBi, but the basic insight holds: you have to tell the model about order somehow, because attention by itself genuinely does not care.⁸

The Encoder Layer

Each encoder layer has two sub-layers:

Multi-head self-attention, where every input token attends to every other input token.
A position-wise feed-forward network, two linear transformations with a ReLU (or GeLU) in between, applied independently to each position.

Each sub-layer is wrapped in a residual connection followed by layer normalization.

The division of labor here is worth internalizing. Attention routes information between positions. The feed-forward network processes it, the same small network run separately at every position. There’s good evidence that the feed-forward layers are where a lot of the model’s factual knowledge actually lives.⁹

The residual connections aren’t cosmetic. They create a gradient highway that lets the training signal flow backward through many layers without vanishing, which is the exact failure that crippled deep RNNs. The idea is borrowed wholesale from ResNets, and it’s a big reason you can stack these layers dozens deep and still train them.

The Decoder Layer

The decoder adds one component the encoder doesn’t have: cross-attention. Sitting between its self-attention and its feed-forward block, the decoder attends into the encoder’s output. The queries come from the decoder, the keys and values from the encoder. This is the literal pipe through which information flows from the input sentence to the output sentence.

The decoder’s self-attention is also masked: each position can only attend to positions at or before it, never ahead. This is what stops the model from cheating by peeking at the answer during training.¹⁰

The Complete Encoder-Decoder Data Flow

If you want the whole forward pass in order:

Input embedding. Convert input tokens into \(d_{model}\)-dimensional vectors.
Add positional encoding. Inject order.
Encoder stack (\(N=6\)): each layer does self-attention, then add-and-norm, then FFN, then add-and-norm.
Output embedding. Convert the output tokens (shifted right by one) into vectors.
Add positional encoding on the output side too.
Decoder stack (\(N=6\)): masked self-attention, add-and-norm, cross-attention over the encoder output, add-and-norm, FFN, add-and-norm.
Linear projection plus softmax. Map the decoder’s output to a probability distribution over the vocabulary.

The whole thing trains end to end with backpropagation and the Adam optimizer. The original paper used a learning-rate warmup schedule, ramping the rate up linearly for the first few thousand steps and then decaying it as the inverse square root of the step count. The warmup turned out to matter a lot for stability and got adopted nearly everywhere, another of those unglamorous tricks that the architecture quietly depends on.

Why It Actually Won (And It Isn’t Elegance)

Here is where I want to push back on the standard story, including the one I used to tell myself. The usual explanation for the Transformer’s success is a tidy list of architectural virtues, and the list is real, but I’ve become convinced it buries the lede.

The tidy list goes like this. Parallelism: self-attention touches all positions at once, so training parallelizes beautifully across sequence positions. Constant-length paths: any two tokens interact in one attention step instead of \(n\) recurrent ones, so long-range dependencies are easy to learn. Compositional computation: stacking attention and feed-forward layers gives a flexible computational model that can apparently implement a surprising range of algorithms, not just pattern matching. Smooth scaling: as you grow model size, data, and compute, performance improves as a clean power law, and that predictability is what justified pouring billions into ever-larger models.

All true. But notice that the first item on the list quietly carries the other three. The reason I now think the Transformer won is less “it has a superior inductive bias” and more “it is the architecture that best exploits the hardware we happened to have.” This isn’t my contrarian hunch, it’s something the people who lived through the transition keep saying out loud.

The HN commenter HarHarVeryFunny, who clearly worked in the area, put it about as bluntly as you can: “Without fast parallel hardware there would neither have been the incentive to design the Transformer, or much benefit even if someone had come up with the design all the same. The incentive came from language model researchers who had been working with recurrent models such as LSTMs, whose recurrent nature made them inefficient to train, and wanted to come up with a new seq-2-seq model that could take advantage of the parallel hardware that now existed and, since AlexNet, was now being used to good effect.”

Read that twice. The Transformer was not a search for the most beautiful model of language. It was a search for a model of language that a GPU could chew through fast. The architecture is shaped like the hardware it runs on. That is also why the original paper’s headline result was so unglamorous: it was about 3.5 times faster to train than the previous state of the art on the same hardware while matching or beating its quality. The revolution arrived disguised as an efficiency win.

I think this matters because it reframes the whole scaling story. If the architecture that wins is the one that parallelizes best, because that’s the one you can afford to make enormous, then the Transformer’s dominance is partly a statement about NVIDIA’s product line and not only about attention being a deep truth about cognition. The commenter crlang44 captured the dynamic from the other side: it wasn’t “advances on the underlying operation of matrix multiplication that have driven AI progress,” it was “the layers above that, trying different neural architectures.” The matmul was always there. The Transformer is the architecture that turned an ocean of cheap matrix multiplication into a language model.

What the Transformer Gets Wrong

No architecture is free, and the Transformer’s bill comes due in a few specific places.

Quadratic attention cost. Self-attention is \(O(n^2)\) in the sequence length, because every token attends to every other token. For a 100,000-token context window, that’s on the order of 10 billion attention scores per layer. This is the single most-attacked weakness in the whole design, and it’s why so much research energy goes into “efficient attention.” The commenter pama gave the cleanest one-liner I found: “Attention is quadratic with context length; RNN with gating (LSTM, GRU, etc) are linear, as are all these new architectures.”¹¹

No inherent recursion. A Transformer has a fixed depth. It cannot natively decide to “think longer” about a harder problem the way you might stew on a puzzle. Chain-of-thought prompting is essentially a hack around this: you externalize the recursion into the token stream itself, letting the model use its own output as a scratchpad. It works startlingly well, which is either reassuring or unsettling depending on your mood.

Position generalization. Despite all the work on positional encodings, Transformers still struggle to generalize cleanly to sequences much longer than they saw in training. This is an active and unsolved area, and it’s why “context window” numbers in press releases deserve a raised eyebrow until you see how performance actually holds up out at the edges.

We don’t really know why it works. This is the one that nags at me most. We have mechanistic interpretability, we have sparse autoencoders, we have induction heads, and they are genuine progress. But the honest summary is still that the gap between “it works incredibly well” and “we understand why it works incredibly well” is enormous. We built the thing, scaled it, and are now reverse-engineering our own creation. The commenter HarHarVeryFunny again, on the broader field: the Transformer “was only ever designed to be a better seq-2-seq architecture, so ‘all you need’ implicitly means ‘all you need for seq-2-seq,’ not all you need for AGI.” We are well past the problem it was designed for, out in territory nobody mapped in advance.

The Variants Zoo

The original Transformer spawned an enormous family, and the family tree tells you something about what survived contact with reality.

Encoder-Only, Decoder-Only, and Encoder-Decoder

Encoder-only (BERT, RoBERTa): bidirectional self-attention, trained with masked language modeling. Great at understanding tasks like classification, named-entity recognition, and semantic similarity.
Decoder-only (GPT, LLaMA, Claude): causal masked self-attention, trained autoregressively. Great at generation. This is the dominant paradigm for large language models as of 2026.
Encoder-decoder (T5, BART): the original full design, with bidirectional encoding, causal decoding, and cross-attention. Still the natural fit for translation and summarization.

The decoder-only branch won the scaling race, and the reason is mostly simplicity: it’s easier to train, easier to scale, and turns out to be sufficient for an absurdly wide range of tasks once it’s big enough. The HN commenter sigmoid10 made the practical point that “Attention is all you need” is actually a bad place to learn modern LLMs, “because it describes a more complicated encoder-decoder architecture while modern LLMs are decoder only,” and pointed people to the GPT-2 paper instead. The thing that won is a strict subset of the thing that was published.

Beyond the encoder/decoder split, the modifications that actually stuck tend to be the ones that buy efficiency without changing the math:

FlashAttention (Dao et al, 2022): an IO-aware implementation that makes attention dramatically faster by being clever about memory reads and writes, without altering the operation itself. Pure systems engineering, enormous practical payoff.¹²

The Challenger: Is Anything Going to Replace It?

For six years the recurring question in the research community has been some version of “okay, but what comes after the Transformer?” and the honest answer is “not yet, but people are trying very hard.”

The most serious challenge comes from state-space models{.link-annotated-not), especially Mamba. Albert Gu, one of Mamba’s authors, announced it with a deliberately pointed line: “Quadratic attention has been indispensable for information-dense modalities such as language, until now.” Mamba offers linear-time scaling in sequence length, which directly attacks the Transformer’s worst weakness. But the community reaction is telling, and it’s more sober than the headlines. The commenter imtringued argued that “Mamba isn’t really a competitor to transformers,” that its real strength “lies in being a better RNN,” good at things like object permanence over long sequences, while you’d still want quadratic attention to actually process dense input like an image.

The pattern that’s emerging isn’t replacement, it’s blending. Look at Jamba, where, as the commenter cs702 summarized, “one out of every eight transformer blocks applies dot-product attention with quadratic cost, but the other seven out of eight apply a Mamba layer with linear cost.” You keep a little attention for the dense recall it’s uniquely good at and offload the rest to something cheaper. The commenter korbip listed the sheer breadth of the effort, more than a dozen named architectures (DeltaNet, RWKV, Retention, Gated Linear Attention, Griffin, xLSTM, Titans, and on and on), most of them sharing a single linear-recurrence update rule.¹³

But here’s the brutally honest assessment, again from a practitioner. The commenter shawntan, working on RWKV, admitted the core obstacle isn’t technical merit at all: “it’s hard to get people away from the addictive drug that is the easily parallelised transformer.” That phrase has stuck with me. The Transformer’s grip isn’t only that it’s good. It’s that an entire ecosystem of hardware, libraries, kernels, intuitions, and trained engineers has crystallized around it. Even a strictly better architecture has to overcome that inertia, and inertia at this scale is a force of nature.

The Monoculture Worry

So let me end where the footnote at the top promised. Every frontier model in 2026 is a Transformer or a near relative. Every major AI lab is betting on the same basic shape. The commenter zarzavat framed the discomfort well: the architecture “has stayed roughly the same since the original AIAYN transformer in 2017,” and “the lack of architecture changes over the last 6 years creates a huge amount of potential energy.” Either we’ve found something close to a local optimum that’s genuinely hard to beat, or we’ve all piled into one design because that’s where the tooling and the talent and the compute already are, and a better idea is sitting just outside the spotlight where nobody’s funded to look.

I don’t know which it is, and I’m suspicious of anyone who claims to. What I keep coming back to is that the Transformer is, in retrospect, the bridge between “AI as a research curiosity” and “AI as an industrial force.” Its parallelism made scaling possible. Scaling made emergent abilities possible. Emergent abilities made the public pay attention. The entire large language model era, GPT-3, ChatGPT, Claude, Gemini, is downstream of one architecture that eight people built to translate German a bit faster.

Understanding it deeply isn’t optional anymore. It’s the prerequisite for understanding anything else in modern AI, including, eventually, whatever replaces it.