AlexNet and the moment deep learning stopped being theoretical
The second-place team at the 2012 ImageNet Large Scale Visual Recognition Challenge scored a top-5 error rate of 26.2 percent. The first-place team, three people at the University of Toronto submitting under the name SuperVision, scored 15.3. In a field that had been gaining fractions of a percentage point per year, that gap was a cliff edge.
The contest measured a simple thing: given a photograph, does your system name the correct category somewhere in its top five guesses? It was hard enough that in 2010 the best approach scored 28.2 percent wrong. By 2012, years of careful refinement had pulled that to 25.8. Then team SuperVision submitted its results on September 30, 2012, and the conversation changed.
The three people behind the entry were Alex Krizhevsky, a Ukrainian-born PhD student who had spent two years teaching himself GPU programming; Ilya Sutskever, then finishing his doctorate; and their supervisor, Geoffrey Hinton, who had been arguing since the 1980s that networks modeled loosely on the brain were the right architecture for machine intelligence — mostly to a field that was politely unconvinced. For the contest, Krizhevsky built a convolutional neural network eight layers deep, with 60 million parameters, and trained it on 1.2 million labeled photographs from Fei-Fei Li’s ImageNet dataset, running on two consumer Nvidia GTX 580 graphics cards. The network trained for five to six days. It beat every hand-crafted system in the field.
The images were Fei-Fei Li’s project. Beginning around 2007 at Princeton, she had argued that the constraint in computer vision was not the algorithms but the data. She built ImageNet around WordNet’s noun hierarchy and crowdsourced the labeling through Amazon Mechanical Turk, recruiting 49,000 workers in 167 countries to verify 160 million candidate photographs. The dataset that emerged — presented at CVPR in 2009 — held 14 million images across 22,000 categories. Without it, AlexNet is a clever idea that trains on nothing.
Three months after the contest closed, Hinton was standing at a laptop in a hotel room at Harrah’s in Lake Tahoe, watching Gmail refresh. It was December 2012, and the neural information processing systems conference was running downstairs. Hinton stood because a chronic back problem meant he could no longer sit. On screen: live bids from Google, Microsoft, Baidu, and DeepMind for DNNresearch Inc., the startup he had just incorporated with Krizhevsky and Sutskever. Each company had one hour to raise its bid by at least a million dollars. Microsoft complained that a fellow bidder — Google — had the ability to read the bids, since everyone was using Gmail. The auction closed at $44 million. Hinton later said Google wasn’t the highest bidder; he chose them because it seemed like the environment where he could keep doing work that mattered.
Within two years, nearly every competitive computer vision paper used deep learning. Yann LeCun called the AlexNet result “a turning point for AI” when it was presented at a conference in Florence. The paper has since been cited over 172,000 times. The field had learned that scale — data and compute — was the dominant variable, and that the right architecture, given enough of both, would learn the features itself.
Hinton once summarized what each of the three had contributed: “Ilya thought we should do it, Alex made it work, and I got the Nobel Prize.” He meant it as a joke. The Nobel committee, in 2024, did not.
Sources
- AlexNet — Wikipedia — Team composition, submission date (September 30, 2012), top-5 error rates, GPU hardware, training duration, architecture, DNNresearch formation and sale.
- CHM Releases AlexNet Source Code — Computer History Museum — Historical significance, Yann LeCun quote, citation count, shift from hand-crafted to learned features.
- ImageNet — Wikipedia — Fei-Fei Li’s role, Mechanical Turk crowdsourcing, 49,000 workers across 167 countries, 2009 CVPR release, dataset scale.
- Google Scoops Up Neural Networks Startup DNNresearch — TechCrunch — Acquisition by Google for $44 million, March 2013.