BERT reads the whole sentence
In October 2019, a Google engineer pointed to a single search query to explain the difference the model had made: “2019 brazil traveler to usa need a visa?” The previous system, reading left to right, weighted “Brazil” as the destination and returned visa rules for people entering Brazil. The correct answer was for travelers leaving it — and it turned entirely on the preposition “to.” BERT, deployed in Google Search that month, caught the distinction. It had read the sentence in both directions at once.
Twelve months earlier, on October 11, 2018, Jacob Devlin of Google AI Language had posted a paper to arXiv alongside colleagues Ming-Wei Chang, Kenton Lee, and Kristina Toutanova: “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” The transformer architecture had existed for barely a year — Vaswani et al. had published “Attention Is All You Need” in June 2017 — and most researchers using it had followed GPT in running it left to right. Devlin’s team ran it sideways.
Language models trained in one direction have a structural flaw: a word is predicted from the words before it, which means reading both directions simultaneously would let the model peek at the very word it is meant to guess. Devlin’s fix was to remove the word before the model ever saw it. During pre-training, 15 percent of words in each input were replaced with a [MASK] token, and the model had to reconstruct them from whatever context remained on both sides. There was nothing to peek at. The model was forced to read.
The model that mastered this trick was sizable by 2018 standards: 24 transformer layers, 340 million parameters, pre-trained on 3.3 billion words from the BooksCorpus and English Wikipedia. When the team benchmarked it against GLUE — nine language-understanding tasks covering entailment, sentiment, and similarity — BERT improved on the previous best score by 7.7 percentage points. On SQuAD 1.1, a question-answering benchmark built from Wikipedia, it scored 93.2 F1, clearing the human-performance mark of 91.2.
Google released the model and its pre-trained weights on November 2, 2018, under the Apache 2.0 license. Fine-tuning it for a new task took hours on a single GPU, not weeks on a cluster — a compression so significant that teams that had never trained a large model from scratch could now compete with those that had. What followed came to be called “BERTology”: a cascade of papers analyzing, extending, and arguing with the model, accumulating more than 60,000 citations in seven years.
What BERT actually established was a workflow. Before it, building a competitive NLP system meant training a large, task-specific model from the ground up. After it, the standard move was to take one pre-trained foundation and fine-tune it on your problem — in hours, on hardware you already owned. This looks obvious now only because BERT made it look obvious. The preposition “to” had always been there, carrying its full meaning. It took a model trained to read from both ends to notice.
Sources
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — arXiv — original paper by Devlin, Chang, Lee, and Toutanova; architecture, benchmarks, and October 11, 2018 submission date.
- Open Sourcing BERT — Google Research Blog — November 2, 2018 open-source announcement; fine-tuning requirements and GPU details.
- Google BERT Sets New Standards in 11 Language Tasks — Synced — GLUE and SQuAD benchmark results at launch; human-level performance comparison.