Attention is all you need
Early in 2017, Google Brain researcher Jakob Uszkoreit kept returning to the same argument: throw away the recurrent neural network. The architecture everyone in the field had built language systems on for years was unnecessary. A single mechanism called attention could handle everything — if you ran the whole sequence at once, in parallel. His colleagues found this implausible. So did his father, Hans Uszkoreit, one of Germany’s most prominent computational linguists (Wikipedia).
On June 12, 2017, Uszkoreit and seven colleagues at Google — Ashish Vaswani, Noam Shazeer, Niki Parmar, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin — posted their proof to arXiv. They called it “Attention Is All You Need.” The title was the argument.
The problem they were solving had been grinding away since the 1980s. Machines processed language the way a careful reader traces a line with a finger — left to right, one token at a time, each word’s meaning carried forward in a running summary called a hidden state. Recurrent neural networks formalized this, and LSTMs refined it. The trouble was that by the end of a long sentence, the beginning had been compressed into a blurred residue. Worse, the sequential design meant no amount of additional hardware could parallelize training past token-by-token serialization.
Self-attention abolished the chain. Rather than reading left to right, the Transformer looked at every position in a sequence simultaneously and computed, for each token, how much weight to assign every other token. A word like “bank” could attend directly to “river” three positions back or “account” five positions ahead — no intermediate hops, no vanishing context. The entire sequence was assembled in parallel, in a single pass. The team trained the full model in about 3.5 days on eight P100 GPUs; prior state-of-the-art systems had required weeks on comparable hardware (NeurIPS 2017).
The author list carried a small structural signal. All eight names were listed as equal contributors, their order determined by random draw. First authorship in academic machine learning is valuable real estate; a lottery instead of a hierarchy was a polite statement that no one needed to fight, because the result made the argument for them. The Transformer beat every prior system on English-to-German and English-to-French translation, topping previous best ensemble results with a single model.
When the paper reached NeurIPS in December 2017, the field moved fast. Within a year, Jacob Devlin at Google had pre-trained a stack of Transformer encoders on the entirety of English Wikipedia and called it BERT. Within two years, OpenAI had stacked decoders into GPT-2. By 2020, GPT-3 ran on 175 billion parameters. ChatGPT arrived in November 2022. The “T” in GPT — in every large language model now in commercial production — stands for Transformer. Most of the paper’s eight authors have since left Google: Aidan Gomez co-founded Cohere, and others dispersed to competing frontier labs and startups. As of 2026, the paper has accumulated more than 250,000 citations, placing it among the most-cited scientific papers of the 21st century in any field.
The last sentence of the abstract mentioned plans to extend the Transformer to “other tasks such as images, audio and video.” That turned out to be something of an understatement.
Sources
- Wikipedia: Attention Is All You Need — Author affiliations, June 12 2017 arXiv posting, Jakob Uszkoreit’s role and the skeptical reception, citation count.
- Proceedings of NeurIPS 2017: original paper — Translation benchmarks, training setup (8 P100 GPUs, 3.5 days for the big model), architectural description.
- freeCodeCamp: The Paper That Created Modern AI — Post-Google ventures including Cohere; Google Translate as original motivation.