Recurrent Neural Networks: The Architecture That Almost Won

The Network That Eats Its Own Tail

The simplest recurrent network is almost embarrassingly small. It’s usually called an Elman network, after the cognitive scientist Jeff Elman, whose 1990_36ya paper “Finding Structure in Time” is one of those papers that is doing serious science while looking like it’s barely trying. At each time step \(t\), the network computes a hidden state \(h_t\) from the current input \(x_t\) and the previous hidden state \(h_{t-1}\):

\[h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b)\]

Any output you need comes off the hidden state, \(y_t = W_{hy} h_t\). That is the entire machine. The same three weight matrices are reused at every step, which is what lets a single network swallow sequences of any length. It’s the temporal cousin of the weight sharing that makes convolutional networks work, and it’s the source of both the architecture’s elegance and its central problem.²

Here is the part that took me a while to feel in my gut. When you “unroll” a recurrent network across a sequence of length \(T\), you get a feedforward network that is \(T\) layers deep, where every layer shares the same weights. Training it means running backpropagation through that whole unrolled stack, a procedure with the wonderfully literal name backpropagation through time. So a recurrent network processing a hundred-word sentence is, for the purpose of training, a hundred-layer-deep network. And deep networks, as anyone who lived through the pre-ResNet era will tell you, were miserable to train. The recurrence didn’t avoid the problems of depth. It manufactured depth, on demand, proportional to your sequence length.

Why the Memory Leaked

The fatal flaw was diagnosed early, and independently, by Sepp Hochreiter in his 1991_35ya diploma thesis and by Yoshua Bengio and colleagues in a 1994_32ya paper. It’s called the vanishing gradient problem, and once you see the math it’s hard to unsee. The gradient that tells the network how an early input should have changed depends on a product of Jacobians, one per time step:

\[\frac{\partial h_T}{\partial h_t} = \prod_{k=t+1}^{T} \frac{\partial h_k}{\partial h_{k-1}}\]

Each factor is governed by the recurrent weight matrix \(W_{hh}\) and the slope of the activation function. Multiply a number slightly less than one by itself a hundred times and you get essentially zero. Multiply a number slightly bigger than one and you get an explosion. So the gradient from a distant time step either decays to nothing or blows up to infinity, and for sequences longer than maybe ten or twenty steps, one of those two things reliably happens.³

The practical consequence was brutal. A vanilla RNN had an effective memory of roughly ten to twenty time steps. For a lot of language, that’s not enough. The subject of a sentence and its verb can sit fifty words apart. The antecedent of a pronoun can be a paragraph back. A network that forgets everything older than the last clause is not going to understand much, and the painful thing was that the architecture wasn’t wrong about what it needed to do, it was wrong about whether SGD could teach it to do it. The capacity was there. The learning signal wasn’t.

This gap, between what a model can represent and what it can be trained to learn, is one of the most important and least intuitive ideas in deep learning, and recurrent networks are where I first understood it. You can build a network that is theoretically capable of remembering something for a thousand steps and still find it utterly unable to learn to do so. Representation and optimization are different problems. The vanishing gradient is an optimization problem wearing a representation problem’s clothes.

LSTM: a Memory With Locks on It

The fix that mattered came from the same person who had diagnosed the disease. Sepp Hochreiter and his advisor Jürgen Schmidhuber published long short-term memory in 1997_29ya. The name is a small joke that became a serious term of art: ordinary RNNs had a short-term memory that decayed too fast, and the goal was to make that short-term memory long. The trick was to add a second channel, a cell state \(c_t\) that runs straight through the sequence with only gentle, additive, linear interactions, so gradients can travel along it without being repeatedly squashed.⁴

What makes the cell state stay useful is a set of gates: small learned sigmoid units that each output a number between zero and one and act like soft valves on the flow of information. The modern LSTM has three:

1. The forget gate \(f_t\) decides how much of the old cell state to keep. \(f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)\). A value near one says “hold on to this,” near zero says “let it go.”

2. The input gate \(i_t\) decides how much new information to write, where the candidate content is \(\tilde{c}_t = \tanh(W_c \cdot [h_{t-1}, x_t] + b_c)\).

3. The output gate \(o_t\) decides how much of the cell state to reveal as the visible hidden state.

The whole thing comes down to two update rules. The cell state is updated by \(c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t\), and the hidden state is read out as \(h_t = o_t \odot \tanh(c_t)\). The first equation is the one that matters. Notice that the old cell state \(c_{t-1}\) is carried forward by multiplication with the forget gate and addition of the new content, with no nonlinearity sitting on the path. If the network learns to hold the forget gate near one, information flows through the cell state nearly unchanged for hundreds of steps, and so do the gradients. The vanishing gradient is defeated not by removing the depth but by building a clean highway through it.⁵

There’s a detail here that I think is genuinely funny: the original 1997_29ya LSTM didn’t have a forget gate. The network could write to and read from its cell, but it couldn’t deliberately clear it, which meant the cell state could saturate and the network had no way to say “this memory is stale, dump it.” The forget gate was bolted on three years later by Gers, Schmidhuber, and Cummins (2000_26ya), and it turned out to be so important that the version everyone now calls “the LSTM” is really the year-2000 revision. The thing that finally made long memory work was, fittingly, learning how to forget.⁶

The credit for all this is, characteristically for deep learning, a minor war. Schmidhuber has spent years arguing, with some justice and a great deal of persistence, that his lab’s contributions (LSTM among them) are systematically under-credited in favor of the more famous trio of Bengio, Hinton, and LeCun. One HN commenter summarized the partisan version of the field’s founding myth as a list of camps, with “The Conspirators: Hinton, Lecun, Bengio et al” on one side and “The Swiss School: Schmidhuber et al, LSTMs as a path to general AI” on the other. Another, logicchains, went further and claimed “there’s a deliberate movement among some people in the ML community to deny Jürgen Schmidhuber credit for inventing LSTMs and GANs.” I don’t have a dog in that fight, but the bare facts are not really in dispute: the LSTM came out of Schmidhuber’s lab in Switzerland, it was developed, as the commenter dharma1 noted, “on European taxpayer’s dime,” and for about a decade almost nobody outside a few speech and handwriting groups cared. When the deep learning boom arrived, it turned out one of the most commercially important architectures in the world had been sitting in plain sight since the Clinton administration.

The Golden Age, 2013_13ya To 2017

For about four years, gated RNNs looked unstoppable. This is the part of the story that’s easy to forget now, because the Transformer’s victory was so total that it retroactively made the LSTM era look like a prelude. It didn’t feel like a prelude at the time. It felt like arrival.

The flagship application was machine translation. Sutskever, Vinyals, and Le (2014_12ya) showed that a deep LSTM could read an entire source sentence into a fixed vector and then generate the translation from it, and the resulting sequence-to-sequence framework became the template for a generation of NLP systems. Google Translate switched to a neural LSTM-based system in 2016 and the quality jump was large enough to make the news. Speech recognition went the same way: Google’s voice search moved to LSTM acoustic models in 2015_11ya and cut word error rates substantially. LSTMs captioned images in Show and Tell, they were generating handwriting in Graves’s beautiful 2013_13ya work, and the DeepMind AlphaStar agent that beat professional StarCraft II players a few years later still leaned on LSTMs in its core.⁸

The cultural high-water mark, though, wasn’t a product. It was a blog post. In May 2015_11ya Andrej Karpathy published “The Unreasonable Effectiveness of Recurrent Neural Networks”, and it did more to popularize what these networks could do than any paper. He trained small character-level RNNs on raw text and let them babble: fake Shakespeare, fake Wikipedia markup, fake LaTeX that almost compiled, fake C source code complete with plausible comments and balanced braces. The output was wrong in the specifics and uncannily right in the texture, and that gap was the whole point. The post still gets cited as an origin story. One commenter on a thread years later put the lineage explicitly: Karpathy’s post “inspired Alec Radford at OpenAI to do the research that produced the ‘Unsupervised sentiment neuron,’” and OpenAI then “decided to see what happened if they scaled up that model by leveraging the new Transformer architecture invented at Google, and they created something called GPT.” The road to GPT runs straight through a char-RNN generating bad Shakespeare.⁹

The piece that pushed RNNs to their peak, and ironically also planted the seed of their replacement, was attention. Bahdanau et al (2014_12ya) noticed that cramming an entire source sentence into one fixed-size vector was a bottleneck, and that the decoder would do better if it could look back at all of the encoder’s hidden states and pull out whichever were relevant at each step. Attention-augmented LSTMs became the dominant paradigm for sequence tasks from roughly 2015_11ya to 2017. The bottleneck was patched. And then someone asked the dangerous question: if attention is doing the heavy lifting, what is the recurrence even for?

Why the Transformer Won, and Why It Isn’t about Elegance

In 2017, Vaswani et al published “Attention Is All You Need”, and the title was a thesis statement. Take the attention mechanism that had been bolted onto LSTMs, throw away the recurrent spine it was attached to, and see what’s left. What was left turned out to be enough for almost everything. I’ve written about the Transformer’s internals elsewhere; here I only care about why it beat the RNN, and the standard answer, repeated in every tutorial, is a clean list of three advantages.

Parallelism. An RNN computes \(h_t\) from \(h_{t-1}\), so the hundredth step cannot start until the ninety-ninth finishes. The whole sequence is a chain of dependencies. A Transformer computes all positions at once, as a matrix multiplication, so a GPU can chew through an entire sequence in parallel. Path length. In an RNN, information from position one reaches position one hundred only by passing through ninety-nine intermediate hidden states, decaying a little each time even with gates; in a Transformer, position one attends to position one hundred directly, an \(O(1)\) path instead of \(O(n)\). Scalability. Because training parallelizes, you can pour more hardware at a Transformer and it keeps getting better in the smooth, predictable way the scaling laws describe, which is what justified the enormous bets that produced GPT-3 and its successors.

All three are true. But I’ve come to think the list buries the real story, which is that the first item silently contains the other two, and the first item is about hardware, not about language. The Transformer didn’t win because attention is a deeper truth about how meaning works. It won because attention is the operation that the GPUs we happened to have could run fastest.

This is not my contrarian reading. It’s what the people who lived through the transition keep saying. One HN commenter, HarHarVeryFunny, who clearly worked in the area, laid it out about as plainly as possible: “Without fast parallel hardware there would neither have been the incentive to design the Transformer, or much benefit even if someone had come up with the design all the same. The incentive came from language model researchers who had been working with recurrent models such as LSTMs, whose recurrent nature made them inefficient to train, and wanted to come up with a new seq-2-seq model that could take advantage of the parallel hardware that now existed.” Read that twice. The motivating problem wasn’t “LSTMs don’t understand language well enough.” It was “LSTMs are inefficient to train on the hardware we now have, because backpropagation through time is inherently serial.” The Transformer is shaped like the GPU it runs on.

You can see this in the original paper’s own framing. Its headline result wasn’t “we understand translation better.” It was that the model reached comparable or better quality while being substantially faster to train, because it parallelized where the LSTM couldn’t. The revolution arrived disguised as an efficiency win, which is exactly what you’d expect if the binding constraint was compute all along. And once you accept that framing, the scaling story reads differently: the architecture that wins is the one you can afford to make enormous, and the one you can afford to make enormous is the one that parallelizes. Attention being a profound model of cognition is, at most, a happy bonus on top of attention being a matrix multiplication a tensor core loves.¹⁰

The transition, once it started, was fast and merciless. BERT (2018) showed a bidirectional Transformer crushing LSTM models across essentially every NLP benchmark at once. GPT-2 (2019) showed a one-directional Transformer generating text so coherent it spooked its own creators. By 2020 it was rare to see a new NLP paper reaching for an RNN if a Transformer would fit. The architecture that had run Google Translate two years earlier was, for research purposes, over.

The irony at the center of all this still delights me. Attention was invented as a crutch for RNNs, a way to relieve the fixed-vector bottleneck in an LSTM translator. The Transformer’s move was to notice that the crutch could walk on its own and the patient could be left behind. The component added to save recurrence is the component that killed it.

Are RNNs Actually Dead?

Here is where the obituary refuses to hold. Because the Transformer’s great weakness is the mirror image of the RNN’s great strength, and that symmetry has kept a stubborn, interesting research program alive.

The Transformer’s weakness is that self-attention is quadratic. Every token attends to every other token, so cost grows as the square of the sequence length, and a long context becomes punishingly expensive. An RNN, by contrast, processes each token in constant time and constant memory, carrying only its fixed-size state forward; it is linear in sequence length and needs no growing KV cache at inference. The RNN’s problem was never its inference profile, which is lovely. Its problem was that you couldn’t train it in parallel. So the obvious dream, chased hard since about 2020, is an architecture with the RNN’s cheap linear inference and the Transformer’s parallel training at the same time.

The leading candidate is the state-space model, and especially Mamba (Gu and Dao, 2023). State-space models borrow old mathematics from control theory and express the sequence update as a linear recurrence that can be computed as a parallel scan during training and as a plain recurrence during inference. Albert Gu introduced Mamba with a line aimed squarely at the Transformer: quadratic attention, he wrote, “has been indispensable for information-dense modalities such as language, until now.” It scales linearly, it has constant-memory inference, and at moderate scale it matched Transformers on language modeling. The excitement was real.

So was the skepticism, and it was sharper and more useful than the hype. The HN commenter imtringued, who clearly follows the area closely, argued that “Mamba isn’t really a competitor to transformers,” that its real strength “lies in being a better RNN,” good at things like object permanence over long sequences, while you’d still want quadratic attention to process information-dense input directly. That reframing matters. The interesting modern architectures may not be Transformer-killers so much as the RNN, rehabilitated, finally trainable at scale and finding the niches where its linear, stateful nature is genuinely the right tool.

The clearest sign of where this is going is that the winning systems aren’t choosing. They’re blending. Look at Jamba or NVIDIA’s Nemotron line, where, as one commenter described it, “one out of every eight transformer blocks applies dot-product attention with quadratic cost, but the other seven out of eight apply a Mamba layer with linear cost.” You keep a little attention for the dense recall it’s uniquely good at, and you offload the rest to a cheaper recurrent layer. The field of pure alternatives is enormous, RWKV, RetNet, DeltaNet, Griffin, xLSTM (a direct revival of the LSTM, with Hochreiter himself as senior author), most of them variations on a single linear-recurrence idea. There’s even a paper cheekily titled “Were RNNs All We Needed?”¹¹

But I want to end on the most honest thing anyone in those threads said, because it explains why “better on a benchmark” hasn’t translated into “replaces the Transformer.” The commenter shawntan, who actually works on RWKV, named the real obstacle, and it isn’t technical: “it’s hard to get people away from the addictive drug that is the easily parallelised transformer.” Another commenter, writing under the title “The Transformer Attractor,” catalogued the graveyard: “Mamba rewrote its core operations as GEMMs, RetNet was abandoned by its own authors at Microsoft Research, RWKV hit a ceiling at 14B despite years of community effort,” and argued that Transformers and NVIDIA GPUs “co-evolved into a stable attractor basin” defended by two reinforcing gates: can you saturate the tensor cores, and will a major lab bet on you. A new architecture has to clear both, and the gates protect each other.

This is the same lesson the whole RNN story keeps teaching, just from the other side. The LSTM lost not because it was a worse model of language but because it couldn’t exploit the hardware. The modern recurrent revivals struggle not because they’re worse but because an entire ecosystem of kernels, libraries, trained engineers, and institutional habit has crystallized around attention. Architecture doesn’t win on merit alone. It wins on merit times hardware times inertia, and the last two terms are doing more work than anyone likes to admit.

What the Whole Arc Taught Me

The recurrent neural network is, in retrospect, a near miss that mattered enormously. Its central intuition, that sequence understanding wants persistent memory, was good enough to drive a decade of progress and seed the ideas that became modern AI. Its central mechanism, learning that memory by propagating gradients through a serial chain, was the thing that hardware refused to reward.

Three lessons stuck with me. The first is the one about Rich Sutton’s “bitter lesson”: the RNN encoded a strong, human-intuitive inductive bias about how sequences should be processed, one step at a time, and at scale that bias became a cage. The Transformer’s weaker bias let data and compute fill in the structure, and it won the way general compute-leveraging methods keep winning. The second is that the architecture that wins is partly a fact about NVIDIA’s product roadmap and not only about the nature of language, which should make anyone humble about reading deep truths into whichever design happens to be on top. And the third is the one I keep turning over: the recurrent idea was never refuted. It was outrun. And now that we’ve learned to train recurrence in parallel, it’s quietly walking back into the frontier through the side door, wearing the name “state-space model” so nobody panics.

The best architecture is always, in part, a function of the hardware it runs on. If the hardware changes again, neuromorphic chips that are natively serial, say, or anything that makes parallel matrix multiplication less special, I would not bet against the network that eats its own tail coming all the way back.

Further Reading

• The Transformer Architecture, the design that displaced RNNs, explained from the inside

• Attention Is All You Need, my take on the 2017 paper and its context

• Attention Mechanisms, how attention evolved from an RNN crutch into the whole show

• Backpropagation, the training procedure that vanishing gradients sabotage

• The Scaling Hypothesis and Neural Scaling Laws, why parallel-friendly architectures got to go big

• Convolutional Neural Networks, the other architecture whose strong bias was loosened at scale

• Andrej Karpathy, “The Unreasonable Effectiveness of Recurrent Neural Networks”, the post that made RNNs famous

• Christopher Olah, “Understanding LSTM Networks”, still the clearest visual explainer of the gates

This piece draws on the original LSTM paper (Hochreiter and Schmidhuber, 1997_29ya) and the forget-gate revision (Gers et al, 2000_26ya); Elman (1990_36ya); Bengio et al (1994_32ya) and Pascanu et al (2013_13ya) on gradients; Cho et al (2014_12ya) on GRUs; Sutskever et al (2014_12ya) on seq2seq; Bahdanau et al (2014_12ya) on attention; Vaswani et al (2017) on the Transformer; the Mamba paper (Gu and Dao, 2023) and RWKV; and a range of practitioner discussion on Hacker News. Diagrams are from Wikimedia Commons under the licenses noted in each caption.