Supervised Learning: Linear/Logistic Regression, Decision Trees & SVM

Explanation

Supervised learning algorithms formulate learning as cost function optimization. In Linear Regression, the cost function J(w) is convex, guaranteeing convergence to the global minimum via Gradient Descent: w := w - alpha * grad(J). In Logistic Regression, the non-linear sigmoid activation compresses unbounded real numbers into [0, 1], representing class membership probabilities. Support Vector Machines find the optimal separating hyperplane that maximizes the margin 2 / ||w||, solving a quadratic programming optimization problem. For non-linearly separable data, Mercer kernels project features implicitly into infinite-dimensional Hilbert spaces without calculating higher-dimensional coordinates directly.

Key points

• Logistic Regression maps linear combinations to probabilities using the Sigmoid function: sigma(z) = 1 / (1 + e^(-z)); optimized using Binary Cross-Entropy loss.
• Decision Trees (ID3, C4.5, CART) recursively partition feature space based on Information Gain (Entropy) or Gini Impurity.
• Support Vector Machines (SVM) maximize the geometric margin between decision boundary and support vectors; employs the Kernel Trick (RBF, Polynomial) for non-linear data.
• Naive Bayes classifier applies Bayes' Theorem with the strong assumption of class-conditional feature independence.

Example

Code examples

import numpy as np

# Gradient Descent for Simple Linear Regression
def gradient_descent(X, y, alpha=0.01, epochs=1000):
    m = len(y)
    w = 0.0
    b = 0.0
    for _ in range(epochs):
        y_pred = w * X + b
        dw = (-2/m) * np.sum(X * (y - y_pred))
        db = (-2/m) * np.sum(y - y_pred)
        w -= alpha * dw
        b -= alpha * db
    return w, b

Exam questions

• [10 Marks] Derive the Gradient Descent update equations for Linear Regression with Mean Squared Error loss.
• [10 Marks] Explain the working of Support Vector Machines (SVM). How does the Kernel Trick enable linear algorithms to separate non-linear datasets?