BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

The World Before BERT

In late 2018 the NLP world was split into two rough camps, and both of them had the same blind spot.

The first camp built task-specific models. If you wanted to do sentiment analysis, you designed an architecture for sentiment analysis. If you wanted named-entity recognition, you bolted a conditional random field onto an LSTM and trained it from scratch on whatever labeled data you had. Every problem got its own bespoke pipeline, its own inductive biases, its own paper. It was a craft, and like most crafts it didn’t scale.

The second camp was the early transfer-learning crowd, and they had the better idea. ELMo (Peters et al, 2018) and OpenAI’s GPT had both shown that you could pre-train a language model on a large unlabeled corpus and then fine-tune it for downstream tasks, and that this beat training from scratch.¹ This was the right direction. But both had a limitation that, once you see it, you can’t unsee.

ELMo used a bidirectional LSTM, but its bidirectionality was shallow: it trained a left-to-right model and a right-to-left model independently and then glued their outputs together. The two halves never actually talked to each other during training. GPT was a Transformer, which was more modern, but it used only the decoder with causal masking, so each word could only see the words before it. Both models, for different reasons, refused to let a word look at its own future.

That sounds reasonable until you think about how reading works. Take the sentence “The animal didn’t cross the street because it was too tired.” What does “it” refer to? You cannot know from the left context alone; you need “too tired” to resolve that “it” is the animal and not the street. Humans integrate context from both directions at once, at every level of processing, without thinking about it. A model that reads strictly left to right is solving a harder version of the problem than it needs to, and a model whose two directions never interact is leaving information on the table. Both camps had accepted this handicap as a law of nature. It wasn’t.

The Trick: Masked Language Modeling

Here’s the circularity that had everyone stuck. If you train a model to predict the next word and you let it see future words, it just reads the answer off the input. That’s not learning, that’s cheating. Left-to-right language modeling avoids the cheating by construction, since each position genuinely cannot see what comes after it. The price is that you throw away half the context.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova at Google AI Language solved this with a move so old it predates computers. Instead of predicting the next word, you blank out some words in the middle of the sentence and ask the model to guess what’s missing. Now bidirectionality is free: the model can look in both directions because the thing it’s predicting has been physically removed from the input. There’s nothing to cheat off of. They called it the Masked Language Model objective, MLM for short, and they were upfront that it’s a direct descendant of the Cloze task from 1950s psycholinguistics, where you measure reading comprehension by deleting words and seeing if people can fill them back in.

The recipe: randomly select 15% of the tokens in each input. Of those, mask most of them and ask the model to reconstruct the originals from the surrounding context. Because a masked token can sit anywhere in the sequence, the model is forced to build representations that fold in information from the left and the right, at every layer, simultaneously. That’s the deep bidirectionality ELMo only faked.

There’s a subtlety the paper handles with a slightly ugly hack, and the hack matters. If you always replace a selected token with the literal [MASK] symbol, you create a mismatch: [MASK] appears constantly during pre-training and never during fine-tuning, because real downstream sentences don’t have holes punched in them. So BERT splits the 15%: roughly 80% get replaced with [MASK], 10% get replaced with a random other word, and 10% are left exactly as they are.² The model never knows which of the three it’s looking at, so it can’t get lazy and only build good representations for the masked positions. It has to keep a useful representation of every token, just in case that token is secretly one of the ones being scored. It’s inelegant, it’s clearly tuned by hand, and it works.

The second pre-training objective was Next Sentence Prediction (NSP). You feed the model two sentences, A and B, and ask it a yes/no question: did B actually follow A in the original document, or did we paste in a random sentence from somewhere else? The motivation was that a lot of downstream tasks, like question answering and natural language inference, are about the relationship between two pieces of text, and the authors wanted BERT to learn something about inter-sentence coherence. Hold onto NSP, because there’s a twist coming: it turned out to be the part of BERT that didn’t really need to be there.

::: {.collapse} ### Deep dive: why masked language modeling works so well

MLM is worth sitting with, because it encodes a genuinely different bet about language than the autoregressive objective GPT uses.

An autoregressive model learns the joint probability of a sentence by factoring it into a chain of conditionals: the probability of the first word, times the probability of the second given the first, times the third given the first two, and so on. This factorization is exact, and it’s also the natural objective if your end goal is generating text, because generation is just sampling from that chain one token at a time. The cost is that it commits you to a single left-to-right reading order.

MLM doesn’t model the full joint distribution at all. It models the probability of each masked token given everything else in the sentence. That’s much closer to a denoising autoencoder than to a classical language model: you corrupt the input by deleting parts of it, and you train the network to undo the corruption. For understanding tasks, classification, extraction, retrieval, you don’t need to generate anything. You need rich, context-saturated representations of the words that are already there. MLM hands you exactly that.

The objective also teaches every level of linguistic structure at once, which is easy to miss. To fill in a masked noun, the model needs syntax (what part of speech fits in this slot?), semantics (what meaning is coherent with the rest of the sentence?), and sometimes plain world knowledge (which named entity actually makes sense here?). A single masked token can require all three at once. That makes MLM an unusually information-dense training signal per example, even though, as the critics correctly pointed out, it’s information-sparse per token.

That sparsity is the real cost. Because only 15% of tokens get predicted on each pass, MLM extracts less learning signal per sentence than an autoregressive model, which predicts every single token. Yang et al made this critique the centerpiece of XLNet, which used a permutation-based objective to get bidirectional context without ever masking anything. One HN commenter put the competitive dynamic plainly: “a chinese student has outperformed and obsoleted BERT (the Google best NLP model) with XLNet. Conversely XLNet wouldn’t have existed without the BERT architecture.”³ In practice the field mostly fixed MLM’s inefficiency the lazy way, by training on more data for more steps, which is the “just scale it” reflex that would come to dominate everything.

Architecture and Scale

Architecturally, BERT is the least surprising part of the paper, which is rather the point. It is just the encoder stack of the original Transformer (Vaswani et al, 2017), with the decoder thrown away. No new attention mechanism, no new normalization scheme, nothing exotic. The paper shipped two sizes:

• BERT-Base: 12 layers, 768 hidden dimensions, 12 attention heads, about 110 million parameters.

• BERT-Large: 24 layers, 1024 hidden dimensions, 16 attention heads, about 340 million parameters.

BERT-Base was deliberately sized to match OpenAI’s GPT at 110M parameters, so the comparison between the two would be apples to apples and the only difference would be the training objective. That’s a tidy bit of experimental design: it isolates the variable the authors actually cared about. BERT-Large was the “and what if we just make it bigger” model.⁴

Both models were pre-trained on the BooksCorpus (800 million words) plus English Wikipedia (2.5 billion words), roughly 3.3 billion words total. The input representation is where a few necessary details live: BERT uses WordPiece subword tokenization with a 30,000-token vocabulary, so it never sees a truly out-of-vocabulary word, just rarer and rarer subword pieces. To each token’s embedding it adds a segment embedding (so the model can tell sentence A from sentence B for the NSP task) and a learned position embedding (the original Transformer used fixed sinusoids; BERT just learns the positions, which works fine up to its length limit).

Two special tokens do quiet but heavy lifting. A [CLS] token is prepended to every input, and its final-layer representation is meant to serve as a summary of the whole sequence, the thing you hang a classifier off of. A [SEP] token marks the boundary between sentences. These conventions are so standard now that it’s worth remembering someone had to invent them.

The compute bill is the detail that startled me most when I went back to check it, because it cuts against the modern intuition that everything important costs millions. Per the reported figures, training BERT-Base took four days on 4 Cloud TPUs (16 TPU chips), at an estimated cost of around 500 US dollars. BERT-Large took four days on 16 Cloud TPUs. Five hundred dollars to pre-train the model that reset the entire field. The barrier was never really the compute; it was knowing what to train and having the TPU access to do it at all. The fine-tuning was cheaper still: the Wikipedia summary notes that BERT-Large needed about an hour on a single Cloud TPU to fine-tune to state of the art on a downstream task. That asymmetry, expensive-ish pre-training amortized across dirt-cheap fine-tuning, is the entire economic logic of the paradigm.

The Results That Broke NLP

BERT’s numbers were, frankly, rude. It set a new state of the art on eleven NLP tasks at once.

On GLUE, the General Language Understanding Evaluation suite of nine tasks, BERT-Large scored 80.5, a 7.7-point absolute jump over the prior best. In a field where papers fought over tenths of a point, a 7.7-point leap was not an improvement, it was an embarrassment to everyone who’d been fighting over tenths. On SQuAD v1.1, the Stanford question-answering benchmark, BERT hit an F1 of 93.2 and edged past the human baseline of 91.2 for the first time.⁵ On SQuAD v2.0, which adds unanswerable questions to punish models that bluff, it beat the previous best by 5.1 points. On SWAG, a commonsense-inference benchmark, it scored 86.3% against a prior state of the art of 66.0%, a twenty-point gap that made people wonder if the benchmark was even measuring the right thing anymore.

These weren’t the usual incremental gains. In several cases BERT vaulted past years of accumulated, task-specific engineering with a single general recipe and no bespoke architecture at all. The collective reaction in the field is captured perfectly by a 2018 HN comment from thwy12321: “Google recently released an NLP algorithm that basically beats all predecessors, but the algorithm itself is extremely simple. Just lots of data and compute.”⁶ That “extremely simple” was both admiration and a little bit of dismay. A lot of people had built careers on the bespoke-architecture approach, and BERT had just announced that the bespoke era was over.

The Fine-Tuning Revolution

The benchmark numbers got the headlines, but the thing that actually changed working life for NLP practitioners was how easy fine-tuning turned out to be.

The pattern was almost insultingly uniform. For a sentence-classification task, take the [CLS] representation, stick a single linear layer on top, and fine-tune the whole model end to end. For a token-level task like named-entity recognition or part-of-speech tagging, take each token’s representation, put a linear layer on each one. For question answering, add two small heads that predict the start and end of the answer span. That’s it. No task-specific architecture, no hand-engineered features, no CRF decoder, no attention mechanism you had to design yourself. You downloaded the weights, bolted on a tiny head, and trained for one to three epochs, often on a single GPU in under an hour.⁷

The “pre-train once, fine-tune everywhere” idea wasn’t BERT’s invention. GPT did it first, and ULMFiT (Howard and Ruder, early 2018) arguably pioneered the recipe for NLP specifically. But BERT demonstrated it at a scale and generality that made it impossible to file under “interesting research direction.” Within months, essentially every NLP leaderboard was topped by BERT or something descended from it. The HN researcher Al-Khwarizmi, who works outside big tech, described the mood with some ambivalence: BERT-style models “have achieved measurable improvements in various (although not all) classic NLP tasks,” and yet, he admitted, “it has made the field more boring, as creative solutions to problems become outperformed by approaches that just pile up more millions” of parameters and dollars.⁸ That tension, real progress that simultaneously flattened the intellectual landscape, is the honest emotional texture of the post-BERT years. The work got better and less fun at the same time.

The BERT Ecosystem

What happened next was an explosion, and a lot of it was driven by infrastructure rather than ideas.

Google released BERT’s code and weights, but in TensorFlow. The thing that actually put BERT into production at thousands of companies was Hugging Face’s PyTorch reimplementation, originally named pytorch-pretrained-bert and later generalized into the Transformers library.⁹ A clean API and a model hub turned “state-of-the-art NLP” from a research artifact into a pip install. I cannot overstate how much this mattered. The HN comment threads from those years are full of people listing BERT in their toolkit next to PyTorch and scikit-learn, treating it as a commodity. That commoditization was Hugging Face’s doing as much as Google’s.

Then came the variants, each one prodding at a different soft spot in the original:

• RoBERTa (Facebook, 2019) made the most quietly devastating point: BERT was undertrained. Train it longer, on more data, with bigger batches, dynamic masking, and no NSP, and it got substantially better without changing the architecture at all. RoBERTa is BERT with the knobs turned up, and it beat BERT comfortably.

• ALBERT cut the parameter count with factorized embeddings and cross-layer weight sharing, aiming for the same quality at a fraction of the memory.

• DistilBERT used knowledge distillation to compress BERT to 60% of its size while keeping about 97% of its performance, which is roughly what you want for putting a model on a phone.

• SpanBERT masked contiguous spans instead of isolated tokens, which helped on tasks where the unit of meaning is a phrase rather than a word.

• ELECTRA replaced MLM with “replaced-token detection,” training the model to spot which tokens had been swapped out by a small generator. Because it learns from every token rather than just 15%, it was far more sample-efficient, directly attacking MLM’s core weakness.

Then there’s the variant that surprised everyone, multilingual BERT (mBERT), trained on the concatenation of 104 languages’ Wikipedias. The shocking part was zero-shot cross-lingual transfer: fine-tune mBERT on an English task, evaluate it on Chinese or Swahili, and it just works, despite never seeing parallel translations or any explicit cross-lingual alignment signal.¹⁰ It apparently learned shared structure across languages purely from their statistical regularities. That spawned XLM and XLM-RoBERTa, and it remains, to me, one of the more genuinely mysterious empirical results of the era.

This proliferation became an academic subgenre of its own. People started probing trained BERT models to figure out what linguistic knowledge lived in which layers, which attention heads tracked syntax, where coreference got resolved. The field even gave the activity a name: BERTology, the interpretive study of what BERT actually learned. A model whose internals are interesting enough to spawn a named research program is a model that did something real.

BERT in Production: Google Search

The most consequential thing BERT ever did was something most people never noticed.

In October 2019, Google announced that it was using BERT to improve Google Search, calling it the biggest leap forward in five years and saying it would affect about one in ten English-language queries. The improvement was specifically about understanding the intent behind a query, especially the small connective words that carry a lot of meaning and that older keyword-matching systems tended to ignore.

Google’s own example is the one everybody quotes, and it’s genuinely clarifying. Consider the search “2019 brazil traveler to usa need a visa.” The crucial word is “to.” This is a Brazilian traveling to the US, not an American traveling to Brazil, and the direction completely flips which answer is correct. Pre-BERT, as Google put it, “our algorithms wouldn’t understand the importance of this connection, and we returned results about U.S. citizens traveling to Brazil. With BERT, Search is able to grasp this nuance.”¹¹ A word as humble as “to” turns out to be where understanding lives, and BERT was the first system at Google’s scale that reliably got it.

This is the moment BERT stopped being a model and became infrastructure, the boring, load-bearing kind that billions of people lean on daily without knowing its name. The HN user potatoman22 summarized the steady-state cleanly: “For most information retrieval purposes, Google’s NLP algorithms can effectively determine the meaning of your query thanks to BERT.”

There’s a darker companion to that story, though, and I think it’s worth airing because it’s the most common complaint about BERT-in-search. Several HN regulars argued that aggressive semantic matching actively degraded search for expert users. The user allovernow put it sharply: Google’s NLP is “far too aggressive for technical searching,” substituting synonyms based on “some sort of BERT-like encoding” so that precise technical queries get rewritten into vaguer, more commercial ones.¹² I have felt this myself, hunting for an exact error string and being served a pile of approximately-relevant SEO content instead. Better language understanding is not automatically better search. Sometimes you want the machine to take your words literally, and a model trained to grasp intent will confidently override you. BERT made search smarter in aggregate and, for a certain kind of power user, more frustrating in particular.

Limitations and Criticisms

BERT has real flaws, and the field spent years working around them.

The hard 512-token sequence limit is the most practical one. Anything longer than roughly a page has to be truncated or chunked, which makes BERT awkward for long documents and spawned a small industry of long-context variants like Longformer and BigBird. The MLM objective’s sample inefficiency, predicting only 15% of tokens per step, makes pre-training more expensive per unit of learning than autoregressive training, which is exactly what ELECTRA and XLNet set out to fix. And the [MASK] pretrain/fine-tune mismatch, even with the 80/10/10 patch, is inelegant in a way that clearly bugged people, given how many later objectives were designed specifically to avoid it.

The deepest limitation is structural: BERT cannot generate text. It’s an encoder. It builds representations; it doesn’t produce sequences. That was fine in 2018, when the prestige tasks were classification and extraction. It became a serious problem as the center of gravity in NLP shifted toward generation, toward chatbots and code completion and open-ended writing, where you need a decoder that emits tokens one at a time. BERT’s whole design assumes you already have the full input in hand and just want to understand it. The future turned out to be mostly about producing output you don’t have yet.

And there’s the matter of fine-tuning itself looking dated. BERT’s paradigm assumes you collect labeled data and fine-tune a fresh copy of the model for each task. The GPT-3 era introduced in-context learning, where a single frozen model performs new tasks from a few examples in the prompt, no gradient updates at all. The HN commenter screye captured the practitioner’s deflation about where this was heading: GPT-3 “is nothing more than one big transformer. At a technical level, it does nothing impressive, apart from throw money at a problem.”¹³ You can hear, in that grumble, the same “extremely simple” reaction BERT got two years earlier, now turned slightly sour. The trick that thrilled people in 2018 was, by 2020, just the cost of admission.

Why the Other Camp Won

So here is the thing that still nags at me. BERT was better than GPT at understanding. The benchmarks said so, loudly, in 2018. And yet the autoregressive, GPT-style, decoder-only lineage is the one that scaled into GPT-4, Claude, Gemini, and the entire modern LLM world. BERT-style encoders did not. Why did the worse-at-understanding camp win?

A few reasons, and none of them is “BERT was wrong.”¹⁴ Generation is commercially enormous in a way that classification simply isn’t; a model that can write, summarize, and converse has a thousand products attached to it, while a model that classifies sentiment has a feature. Autoregressive training is more sample-efficient, since it learns from every token rather than 15% of them, which matters a lot when you scale to trillions of tokens. The decoder-only architecture is simpler to scale and serve, with no separate encoder to manage. And in-context learning, the ability to pick up a task from the prompt with zero fine-tuning, was an emergent gift of the autoregressive objective that the encoder paradigm never offered. Once models got big enough to learn in-context, the whole “collect labels, fine-tune per task” workflow that BERT institutionalized started to look like extra work.

But the failure to win the frontier doesn’t diminish what BERT proved, because BERT’s real contribution was never the model. It was the paradigm: that you can pre-train a large model on unlabeled text with a self-supervised objective and then adapt it to anything. That idea is now so completely assumed that it’s invisible. Every modern LLM is a pre-trained model adapted to downstream use. BERT didn’t invent that idea, GPT and ULMFiT have priors, but BERT proved it at a scale and with a margin that ended the argument. After BERT, “train from scratch for your task” stopped being a respectable default.

Legacy

You will not deploy BERT-Large in 2026 for a new greenfield project; you’ll reach for something newer, probably a small modern encoder or a hosted LLM depending on the task. BERT’s importance is historical and conceptual, not operational, and that’s the right way to read it.

It nailed down two lessons that have only grown more true. The first is that the training objective can matter more than the architecture. BERT and GPT used the same Transformer bricks. The difference between them, bidirectional masked prediction versus left-to-right generation, was a choice about what to train on, and it produced models good at almost opposite things. We spent the next several years trying to get the best of both, with T5 framing everything as text-to-text, BART combining a bidirectional encoder with an autoregressive decoder, and UniLM trying to unify the objectives outright. The objective was the lever the whole time.

The second lesson is the one BERT shares with the Transformer paper itself: nobody can reliably call which idea will matter while it’s happening. The Transformer was “just another interesting project” inside Google before it rewired the industry. BERT looked, to a lot of smart people, like a clever-but-simple trick, “just lots of data and compute,” before it broke eleven benchmarks and rebuilt how a billion people search. The gap between how these papers were received and what they turned out to mean should make all of us humble about predicting which arXiv preprint this week is the one that matters.

BERT was Google’s answer to a small, almost naive question: what if we pre-trained a Transformer to read in both directions at once? The answer was that you demolish every NLP benchmark in sight, you quietly transform web search for the entire English-speaking internet, and you launch a paradigm that the field never abandons. Not bad for a model whose name is a Sesame Street in-joke and whose entire trick was deleting words and asking a computer to guess them back.

The BERT paper is available at arXiv:1810_216ya.04805 (Devlin, Chang, Lee, and Toutanova, 2018). The original code and pre-trained weights are on GitHub, and the most-used implementation lives in Hugging Face Transformers. For the architecture BERT is built on, see the Transformer architecture and the original “Attention Is All You Need” paper.