What Is a Support Vector Machine and How Does It Work?
A support vector machine separates two classes with the widest possible margin, and only the training points closest to that boundary decide where it falls.
A support vector machine (SVM) is a supervised learning method that sorts data into classes by drawing a boundary between them. Of all the boundaries that separate the classes, it chooses the one with the widest empty band on either side, called the margin. The training points that sit on the edges of that band are the support vectors, and they alone fix where the boundary goes.
What are the parts of an SVM boundary?
Picture two groups of points on a sheet of paper, one marked with circles and the other with crosses. In two dimensions the boundary is a line. In higher dimensions it is a flat surface called a hyperplane. It is written as w·x + b = 0, where w is a vector at right angles to the boundary and b sets its offset from the origin. A new point is assigned to a class by the side of the boundary it falls on.
The margin is the band between two parallel lines that pass through the nearest points of each class. The width of that band is 2 divided by the length of w, so a shorter w gives a wider margin. Training an SVM therefore means making w as short as possible while keeping every training point on the correct side of its margin edge.
| Part | What it does |
|---|---|
| Hyperplane | Separates the classes and gives the decision |
| Margin | The empty band on each side of the hyperplane |
| Support vectors | Training points on the margin edges that fix its position |
| Parameter C | Sets the penalty for points that break the margin |
| Kernel | Changes the shape the boundary can take |
Why are only some points important?
Think of the boundary as a fence built between two crowds. The fence only has to stay clear of the people nearest to it. People standing far back in either crowd could move a little without changing where the fence goes. The support vectors are those nearest people, and they are the only training points that hold the fence up.
This has a practical consequence. Deleting a distant training point leaves the classifier unchanged, but moving a support vector changes it. A trained SVM can therefore be compact, because the decision depends on a stored set of support vectors, often a small fraction of the whole dataset. Unlike k-nearest neighbors, which keeps every example and checks neighbours at prediction time, an SVM keeps only the points that matter.
What happens when the classes overlap?
Real data is rarely cleanly separable. A soft-margin SVM lets some points fall inside the margin or on the wrong side, and it charges a penalty for each violation. The parameter C sets the trade-off. A large C punishes errors heavily and produces a narrow margin that fits the training data closely, which can overfit. A small C tolerates more errors in exchange for a wider, smoother boundary. The soft-margin formulation was published by Corinna Cortes and Vladimir Vapnik in 1995, and tuning C is one way to control overfitting.
How do kernels bend the boundary?
Some datasets cannot be split by a straight line in their original space. One class may form a ring around the other, for example. A kernel function lets the SVM compute similarities as if the points had been mapped into a higher-dimensional space, where a flat boundary can work, without ever building that mapping explicitly. This is the kernel trick. The radial basis function kernel is a common choice, and it turns the boundary into a smooth curve or closed shape in the original space.
How is an SVM trained and tuned?
Training an SVM means solving an optimisation problem: keep w small and keep the total penalty for margin violations low at the same time. The same objective can be written with the hinge loss, a function that is zero for points safely outside the margin and grows linearly for points inside it. That form lets gradient descent methods optimise an SVM, although specialised solvers are more common in practice.
Features should be scaled to comparable ranges, because distances drive the margin. C and the kernel settings are usually chosen with cross-validation, by testing several combinations on held-out data. SVMs work well on medium-sized datasets with many features, such as text classification. Training time grows quickly with the number of examples, so very large datasets often call for other methods, including decision trees and random forests or neural networks.
Tags
classification linear algebra machine learning optimization
Related articles
- k-Nearest Neighbors
- Gradient Descent
- What Is Linear Regression? How Least Squares Works
- The Traveling Salesman Problem: Why the Shortest Route Is Hard
- Open Datasets for Machine Learning: The Essential Landscape
- k-Means Clustering