BERT in Practice: Tokens, Embeddings and Fine-Tuning a Text Classifier

My other BERT post covers why the model mattered. This one is the practical companion: what happens to a sentence on its way through BERT, how to try it in a few lines of Python, and the decisions that make or break a fine-tuned classifier. The running example is a common one in finance: sorting short pieces of text, such as news headlines, into a handful of categories.

Step 1: text becomes WordPiece tokens

BERT does not see words. It sees tokens from a fixed vocabulary of about 30,000 WordPiece units (Devlin et al., 2018). Common words are single tokens. Rarer words are split into pieces, with ## marking a piece that continues the previous one. The official repository’s own example splits the name “Philammon” into phil ##am ##mon (google-research/bert).

This matters more in finance than in general text. Tickers, fund names and product jargon are often rare, so they get chopped into fragments that carry little meaning on their own. Before fine-tuning, it is worth tokenising a sample of your real text and looking at what comes out.

The tokenizer also adds two special tokens: [CLS] at the start, whose final vector is used for classification, and [SEP] at the end of each sentence. Inputs are capped at 512 tokens.

Step 2: tokens become contextual vectors

Each token is mapped to an embedding and passed through a stack of Transformer layers in which every token attends to every other. BERT-Base has 12 layers, a hidden size of 768 and about 110 million parameters; BERT-Large has 24 layers, a hidden size of 1,024 and about 340 million (figures from the paper and repository). The output is one 768-number vector per token for BERT-Base, and each vector reflects the whole sentence around it.

“ECB cuts rates”→[CLS] t₁ t₂ … t_n [SEP]→12 × Transformer→h_[CLS] → label

Step 3: try it before you train anything

The Hugging Face transformers library wraps the original weights (BERT model docs). The quickest way to build intuition is the fill-mask pipeline, which asks BERT to do exactly what it was pre-trained to do:

from transformers import pipeline

fill = pipeline("fill-mask", model="google-bert/bert-base-uncased")
for guess in fill("The central bank decided to [MASK] interest rates."):
    print(f"{guess['token_str']:>10}  {guess['score']:.3f}")

Change the context and watch the guesses change. Swap “central bank” for “the landlord” and see what happens to the verbs. Ten minutes of this teaches more about what “contextual” means than any diagram.

Step 4: fine-tune for your labels

For classification you load the same weights with a fresh output layer and train the whole network on labelled examples. A compact version with the Trainer API:

from datasets import load_dataset
from transformers import (AutoTokenizer, AutoModelForSequenceClassification,
                          DataCollatorWithPadding, Trainer, TrainingArguments)

name = "google-bert/bert-base-uncased"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name, num_labels=3)

# CSVs with a "text" column and an integer "label" column (0, 1, 2)
data = load_dataset("csv", data_files={"train": "train.csv", "test": "test.csv"})
data = data.map(lambda b: tok(b["text"], truncation=True, max_length=64), batched=True)

args = TrainingArguments(output_dir="headline-model", learning_rate=2e-5,
                         per_device_train_batch_size=16, num_train_epochs=3,
                         eval_strategy="epoch")
trainer = Trainer(model=model, args=args,
                  train_dataset=data["train"], eval_dataset=data["test"],
                  data_collator=DataCollatorWithPadding(tok))
trainer.train()

The learning rate, batch size and epoch count sit inside the small grid the BERT authors recommended for fine-tuning, which is a sensible place to start before any tuning of your own.

The decisions that actually matter

Cased or uncased. The uncased model lowercases everything. That is usually fine for sentiment, but if capitalisation carries meaning in your text, as with tickers or the difference between “US” and “us”, try a cased model.

Sequence length. Attention cost grows quickly with length, so do not pad headlines to 512 tokens. Measure the token lengths of your data and pick a cap that covers nearly all of it; 64 is plenty for headlines. For long documents, split into chunks or use a model built for long inputs.

Label quality over model size. Two people labelling the same 200 examples and disagreeing on 20% of them tells you the ceiling for any model. Fix the labelling guide before reaching for BERT-Large.

Honest evaluation. Hold out a later time period, not a random sample. Headlines from the same week share stories and wording, so random splits overstate accuracy. Then read the per-class precision and recall, not just accuracy, especially if one category is rare.

Similarity is a different job. If you want to find similar sentences rather than classify them, raw BERT outputs are a poor fit. The Sentence-BERT authors describe BERT’s construction as unsuitable for semantic similarity search and train a modified network for that purpose (Reimers & Gurevych, 2019).

Domain models. Before fine-tuning a general model on financial text, check whether a domain-adapted one already exists. FinBERT is one example, pre-trained further on financial text for sentiment analysis (Araci, 2019).

When not to use BERT

If a keyword rule gets you 95% of the way, ship the rule. If you have a few hundred labelled examples and need explanations, a TF-IDF and logistic regression baseline is quick to build and easy to inspect, and it tells you how much BERT is really adding. BERT earns its place when wording is varied, context changes meaning, and you have enough labelled data to measure the difference.