The Transformer, or how attention replaced recurrence
The first design document circulating inside Google Brain in early 2017 included a reference to the Transformers franchise — the one with the talking cars. The proposed architecture was called the Transformer because one of its eight engineers, Jakob Uszkoreit, simply liked the sound of the word. This seems like an improbable origin story for a piece of technology that would, within a decade, underlie almost every significant artificial-intelligence system in production.
The paper — “Attention Is All You Need” — was submitted to arXiv on June 12, 2017, by eight researchers at Google Brain and Google Research: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. The author order was randomized, since all eight considered their contributions equal. They were building a machine-translation system. That was the application. The architecture was the discovery.
The problem they were solving had a decade of heavy investment behind it. The gold standard for sequence-to-sequence tasks — translating sentences, summarizing text, answering questions — had been recurrent neural networks, particularly a variant called the long short-term memory network. These processed language one token at a time, left to right, threading a hidden state forward through the sequence like a bead on a wire. The approach worked, but it could not be parallelized across a sentence and tended to lose track of relationships in longer passages. The Transformer replaced all of that with a mechanism called self-attention, which allowed the model to weigh every word in a sentence against every other word simultaneously. No loops. No memory. Just attention.
The results were difficult to dismiss: 28.4 BLEU on the WMT 2014 English-to-German benchmark, more than two points above the previous best, trained in 3.5 days on eight P100 GPUs. The English-to-French score came in at 41.8 BLEU on a single model. Machine translation had been a contested benchmark for years; this was not a marginal gain.
The anecdote that follows the paper around involves what happened to its authors afterward. Google published “Attention Is All You Need,” and then, over the next few years, lost all eight people who wrote it. Noam Shazeer left to co-found Character.AI. Illia Polosukhin left to co-found NEAR Protocol. Ashish Vaswani departed in 2021 to build Adept AI Labs, then Essential AI. The others distributed themselves across Cohere, Anthropic, Sakana AI, Inceptive, and elsewhere. The company that had funded and published the most consequential paper in modern machine learning watched its authors walk out, one by one, and build the competition.
As of 2026, the paper has accumulated more than 150,000 citations, placing it among the ten most-cited papers of the twenty-first century. Every serious language model in production today — BERT, the GPT series, Gemini, Claude — runs on the architecture those eight researchers described in twelve pages. The Transformer did not replace one technique; it replaced the organizing principle of the field.
Within five years, the same attention mechanism appeared in protein-folding models, music generators, image classifiers, and drug-discovery systems. It had stopped being a machine-translation trick. It had become the shape of machine intelligence.
Sources
- “Attention Is All You Need” — Wikipedia — author list, randomized ordering, naming story (the Transformers franchise reference), post-publication exodus, citation count exceeding 150,000.
- “Attention Is All You Need” — arXiv 1706.03762 — submission date, BLEU scores on WMT 2014 English-to-German (28.4) and English-to-French (41.8), training time of 3.5 days on eight P100 GPUs.
- Noam Shazeer — Wikipedia — Shazeer’s departure from Google and co-founding of Character.AI.
- Illia Polosukhin — Wikipedia — Polosukhin’s departure from Google and co-founding of NEAR Protocol.