Backpropagation, 1986: the algorithm that revived neural networks
In 1974, Paul Werbos submitted a doctoral thesis to Harvard and left his best idea in one of its chapters. The title was Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. Buried in it was a complete mathematical derivation of an algorithm that could train multi-layer neural networks — the missing piece that had paralyzed the field since Minsky and Papert’s 1969 critique of the single-layer perceptron. The thesis found almost no readers. The algorithm sat in Cambridge and waited (Wikipedia).
Twelve years later, three researchers sent the same idea to every neural-network laboratory in the world.
“Learning Representations by Back-Propagating Errors,” by David Rumelhart, Geoffrey Hinton, and Ronald J. Williams, appeared in Nature 323 in October 1986 — four pages, pages 533 to 536. Rumelhart was a cognitive psychologist at the University of California, San Diego; he had developed the core idea independently in spring 1982 while thinking about how layered networks might form internal representations. Hinton was at Carnegie Mellon University. Williams was a graduate student at UCSD. The paper arrived alongside Rumelhart and James McClelland’s two-volume Parallel Distributed Processing, released the same year, which became the connective tissue of an entire research community (Wikipedia).
The problem backpropagation solved was easy to state. Minsky and Papert had proved in 1969 that a single-layer perceptron could not learn exclusive-or — a four-case function no straight line could separate. Add a hidden layer and XOR becomes solvable; add more layers and the network can represent almost anything. The obstacle was training those hidden layers: you could measure error at the output, but the hidden neurons had no direct signal telling them how to adjust.
Backpropagation solved this by running the chain rule in reverse. In a forward pass, an input flows through the network and produces an output; the error is measured. Then the error flows back — layer by layer, each weight receiving a signal proportional to its contribution to the mistake. Repeat this thousands of times and the hidden layers gradually self-organize into useful representations. The network learns what to notice.
Hinton was initially unconvinced. He preferred Boltzmann machines, a competing architecture, and came around only in 1983 after Terry Sejnowski ran experiments demonstrating that backpropagation worked on real problems. The paper that would put his name on the algorithm was one he had spent a year arguing against (Wikipedia).
Within three years, Yann LeCun was using backpropagation to train convolutional networks at Bell Labs; by 1989 those networks were reading handwritten zip codes for the US Postal Service (Wikipedia). By 2012, a descendant of the same training method cut the ImageNet top-five error rate from 26 percent to 15 percent with a network called AlexNet, and ended a decade of argument about whether neural networks were worth pursuing (Wikipedia). Every language model trained since runs on a variant of the same four pages.
Leibniz wrote down the chain rule in 1676. Three hundred and ten years later, it became the engine of modern AI.
Sources
- Learning Representations by Back-Propagating Errors — Nature, 1986 — The original four-page paper by Rumelhart, Hinton, and Williams; algorithm description and XOR demonstration.
- David Rumelhart — Wikipedia — UCSD affiliation, spring 1982 development of the idea, PDP books, Hinton’s initial preference for Boltzmann machines, Sejnowski’s 1983 experiments.
- Paul Werbos — Wikipedia — 1974 Harvard thesis, first derivation of backpropagation, limited readership.
- LeNet — Wikipedia — LeCun’s 1989 application of backpropagation to handwritten zip-code recognition at Bell Labs.
- AlexNet — Wikipedia — 2012 ImageNet results, error-rate reduction, and the impact on the field.