Support Vector Machines (SVM)

Support Vector Machines (SVM) are powerful supervised learning models used for classification and regression tasks. They work by finding the optimal hyperplane that separates data points of different classes with the maximum margin.

Why Use SVM?

• Works well in high-dimensional spaces.
• Effective when the number of dimensions exceeds the number of samples.
• Robust to overfitting, especially in high-dimensional space.

1. How SVM Works

SVM tries to find the best boundary (hyperplane) that separates classes in the feature space. It focuses on the data points that are closest to the decision boundary — these are called support vectors. The algorithm then maximizes the margin between these support vectors.

• Linear SVM: Works when the classes are linearly separable.
• Kernel SVM: Used for non-linear separation using kernel tricks like RBF or polynomial.

2. Basic Example (Linear SVM)

from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.metrics import classification_report

# Load dataset
iris = datasets.load_iris()
X = iris.data
y = iris.target

# Binary classification: Class 0 vs Class 1
X = X[y != 2]
y = y[y != 2]

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Train SVM model
model = SVC(kernel='linear')
model.fit(X_train, y_train)

# Predict and evaluate
y_pred = model.predict(X_test)
print(classification_report(y_test, y_pred))

Output-

              precision    recall  f1-score   support

           0       1.00      1.00      1.00        17
           1       1.00      1.00      1.00        13

    accuracy                           1.00        30
   macro avg       1.00      1.00      1.00        30
weighted avg       1.00      1.00      1.00        30

3. Example with Non-linear Kernel (RBF)

# Train with RBF kernel
model_rbf = SVC(kernel='rbf', gamma='scale')
model_rbf.fit(X_train, y_train)

# Predict and evaluate
y_pred_rbf = model_rbf.predict(X_test)
print(classification_report(y_test, y_pred_rbf))

Output-

   accuracy                           1.00        30
   macro avg       1.00      1.00      1.00        30
weighted avg       1.00      1.00      1.00        30

4. Key Parameters of SVC

• kernel: Specifies the kernel type – 'linear', 'poly', 'rbf', 'sigmoid'.
• C: Regularization parameter. Smaller values specify a softer margin.
• gamma: Defines how far the influence of a single training example reaches.

5. When to Use SVM

• High-dimensional datasets (e.g., text classification).
• Small-to-medium-sized datasets.
• Binary classification problems.
• When a clear margin of separation exists.

6. Limitations

• Not suitable for very large datasets due to high training time.
• Less effective when classes overlap significantly.
• Model tuning (kernel, C, gamma) can be complex.

Summary

Support Vector Machines are highly effective for classification problems, especially when the data is high-dimensional or has a clear margin of separation. With kernel tricks, SVM can even classify non-linear datasets.