Why BERT Mattered: From Static Word Vectors to Contextual Language Models

Take the word “bank”. In “the central bank held rates”, “we walked along the river bank” and “Monday is a bank holiday” it means three different things. Anyone working with financial text runs into this constantly: “yield”, “security”, “position” and “call” all change meaning with their neighbours. The short version of why BERT mattered is that it was the model that made handling this kind of context the default, and made it cheap enough for everyone else to use.

Before: one vector per word

Through the mid-2010s, a common starting point for NLP was word embeddings such as word2vec (Mikolov et al., 2013). Each word in the vocabulary gets a single vector, learned from which words tend to appear near it. These vectors captured useful similarity: words used in similar ways end up close together.

The limitation is in the design. “Bank” gets one vector, which is a blend of every sense it was trained on. Whatever model you build on top has to untangle the meaning from the surrounding words by itself, usually with a task-specific architecture and plenty of labelled data.

Static embedding

the central bank held ratesv_bank

along the river bankv_bank

a bank holidayv_bank

Contextual (BERT)

the central bank held ratesv₁

along the river bankv₂

a bank holidayv₃

The pieces that were already there

BERT did not appear from nowhere. In 2017 the Transformer architecture showed that attention, where every token can look directly at every other token, could replace recurrent networks for translation (Vaswani et al., 2017). In early 2018 ELMo produced contextual word vectors from a language model trained in both directions and showed they improved many tasks when added as features (Peters et al., 2018). OpenAI’s first GPT model the same year pre-trained a Transformer on unlabelled text and then fine-tuned it, but, as the BERT paper points out, it read text left to right only.

So contextual representations, Transformers and pre-train-then-fine-tune all existed. What was missing was a way to train a deep Transformer that used context from both sides of every word at once.

What BERT changed

The BERT paper (Devlin et al., 2018) solved the bidirectional problem with a simple training trick. If a model can see the whole sentence, predicting the next word is trivial, so instead BERT hides a random 15% of the input tokens and learns to fill them in. This “masked language model” objective lets every layer use left and right context together. A second objective trained the model to tell whether two sentences actually followed each other.

The bigger shift was in how the model was used afterwards. For a new task you keep the whole pre-trained network, add one small output layer and fine-tune everything on your labelled examples. Sentiment, question answering, entity tagging and sentence-pair tasks all used the same recipe. The paper reported new state-of-the-art results on eleven tasks, including a GLUE score of 80.5%, a 7.7-point absolute improvement, and a SQuAD v1.1 test F1 of 93.2.

Just as important, Google released the code and pre-trained weights (google-research/bert). Teams without the budget to pre-train a model could download one and fine-tune it in hours on a single GPU. That is what turned a strong paper into a turning point.

Why it mattered outside research

Three practical effects stand out to me.

Smaller labelled datasets became workable. Because most of the language knowledge comes from pre-training, a classifier for, say, financial news headlines can reach useful accuracy with far fewer labelled examples than a model trained from scratch would need. Labelling is usually the expensive part of a text project, so this changed what was worth attempting.

One architecture for many tasks. Instead of a bespoke model per problem, teams could standardise on one encoder and swap the head. That simplifies maintenance in the same way that standardising on one SQL dialect does.

Domain versions followed quickly. FinBERT, for example, continued pre-training BERT on financial text and fine-tuned it for financial sentiment analysis (Araci, 2019). The same pattern appeared in legal, biomedical and scientific text.

It also reached ordinary users. In 2019 Google said BERT would help Search better understand one in ten English-language queries in the US (Google, 2019).

Where it stands now

BERT is no longer the frontier. Larger generative models built on the same Transformer foundations now handle tasks that BERT could not, such as writing text and following instructions. BERT also has hard limits: the original models accept at most 512 tokens, so long documents must be split, and they do not generate text.

But BERT-style encoders are still a sensible choice for a lot of everyday work: classifying short texts, tagging entities, or producing embeddings for search and de-duplication. They are small, fast and cheap to run, and their behaviour is easier to evaluate than a general-purpose chatbot’s. If the task is “put this sentence in one of six buckets”, a fine-tuned encoder is often the right tool.

The lasting idea is the one in the diagram above: a word’s representation should depend on its sentence. Almost every NLP system since has been built on that assumption. For the hands-on side, see my companion post on tokens, embeddings and fine-tuning with BERT.