So why SVMs?

Many popular pattern classification techniques, such as K-nearest neighbour (KNN), rely on classifying new patterns in the original feature space chosen during the extraction step. The KNN decision rule essentially classifies an input pattern based on a majority rules vote of the classes of the K neighbour points it is closest to.

Although KNN has been shown to perform well in practice, the traditional argument against the use of unedited KNN is that it requires a large amount of storage space (the entire training dataset), and that the entire set has to be queried for each new pattern to be classified.

In addition, it is worth noting that unedited KNN classification does not take the discriminant boundary between classes into account, but is instead based solely on neighbouring points. Thus, potentially redundant training points away from boundary positions are still used in decision making.

What are SVMs?

SVMs instead represent the original clusters of data via support vectors that maximize the margin between data from different classes. The support vectors are a subset of the original training set.

We will examine the case of linearly separable classes.