Support-vector networks: the margin that held for fifteen years
When Vladimir Vapnik joined AT&T Bell Labs in Holmdel, New Jersey in 1991, the algorithm he had been quietly building for thirty years was finally in the right country.
Vapnik had spent his career at Moscow’s Institute of Control Sciences, where, in the late 1960s, he and his colleague Alexey Chervonenkis developed Vapnik-Chervonenkis (VC) theory — a rigorous mathematical account of how well any learning machine can generalize from the examples it has seen to the ones it hasn’t. Soviet academia had little appetite for international exchange, and the theory remained largely unknown in the West until Vapnik emigrated, at fifty-four, to join Bell Labs. He arrived during an interregnum: neural networks, partially rehabilitated by the backpropagation paper of 1986, were still hard to train and short on the theoretical guarantees that mathematicians like Vapnik trusted.
Working with his Bell Labs colleague Corinna Cortes, Vapnik distilled his ideas into a practical learning machine. The paper they produced, “Support-Vector Networks,” appeared in the journal Machine Learning in September 1995. Its central idea was purely geometric: given a dataset split into two classes, find the hyperplane that separates them with the maximum possible margin — the widest road through the data. The points sitting right on the margin’s edge, the ones that would collapse it if shifted, are the support vectors. Everything else is irrelevant to the decision boundary. A machine that learned to ignore most of what it had seen.
Two technical refinements made the idea practical. The soft margin, developed by Cortes and Vapnik in 1993, allowed controlled misclassifications so the classifier could work on real-world data that didn’t separate cleanly. The kernel trick, introduced by Bernhard Boser, Isabelle Guyon, and Vapnik in 1992, enabled curved decision boundaries at linear cost: instead of explicitly transforming data into a high-dimensional space, the kernel function computed distances as if the transformation had occurred. Curved margins, cheap computation.
The benchmarks rewarded this. On MNIST — the handwritten-digit database that Yann LeCun had assembled as the field’s standard test — SVM variants achieved error rates of 0.8 to 1.1 percent by 1999, competitive with the best neural networks of the day. Thorsten Joachims applied SVMs to text categorization in 1998 and found they outperformed every previous method across a broad range of tasks, required no manual parameter tuning, and behaved consistently across corpora of different sizes and topics.
A parallel development ran alongside. AdaBoost, published by Yoav Freund and Robert Schapire in 1997, showed that weak classifiers — models that barely beat random guessing — could be combined into a strong one by iteratively focusing on the examples they kept getting wrong. Leo Breiman’s Random Forests in 2001 assembled hundreds of decision trees on random subsets of the data and averaged their votes. Both methods shared SVMs’ essential virtues: theoretical grounding, reliable behavior, and consistent wins on benchmarks that neural networks kept losing.
The margin held until 2012, when a convolutional network called AlexNet — trained on GPUs by Krizhevsky, Sutskever, and Hinton in Toronto — finished the ImageNet competition with a 15.3 percent error rate. The second-place entry, built on the methods that had ruled for fifteen years, came in at 26.2 percent. Fifteen years of margin, erased in one afternoon.
Sources
- Vladimir Vapnik — Wikipedia — biography, Soviet career at Moscow’s Institute of Control Sciences, emigration timeline, Bell Labs appointment, VC theory development with Chervonenkis.
- Support-vector networks — Springer Machine Learning — Cortes and Vapnik’s 1995 paper; soft margin and kernel contributions.
- Vladimir Vapnik — The Franklin Institute — career highlights and scope of SVM applications.
- MNIST database — Yann LeCun — 1999 benchmark results including SVM error rates of 0.8–1.1%.