Deep Learning Foundations: Perceptrons, MLP & Backpropagation

Explanation

Deep learning overcomes manual feature engineering by learning hierarchical feature abstractions directly from raw inputs. A single-layer perceptron is constrained to linear decision boundaries. By introducing hidden layers equipped with non-linear activation functions (e.g., ReLU), MLPs become universal function approximators. Forward propagation computes predictions and evaluates loss. Backpropagation propagates the error backward through the network layers, using the chain rule to update weight matrices via stochastic gradient descent: W^[l] := W^[l] - alpha * (dL / dW^[l]).

Key points

• Multilayer Perceptron (MLP): Feedforward neural network with input, hidden, and output layers utilizing non-linear activation functions.
• Activation Functions: Sigmoid (squashes to [0,1], prone to vanishing gradient), Tanh (squashes to [-1,1]), ReLU (f(x) = max(0, x), avoids gradient vanishing).
• Backpropagation: Efficient application of the multivariable calculus Chain Rule to compute partial derivatives of loss with respect to all layer weights.
• Deep learning architectures overview: Convolutional Neural Networks (CNNs for spatial/vision data) and Recurrent Neural Networks (RNNs for sequential data).

Example

Code examples

import numpy as np

# Forward propagation for a 2-layer neural network
def forward_pass(X, W1, b1, W2, b2):
    Z1 = np.dot(X, W1) + b1
    A1 = np.maximum(0, Z1) # ReLU activation
    Z2 = np.dot(A1, W2) + b2
    # Softmax activation for multiclass output
    exp_Z2 = np.exp(Z2 - np.max(Z2, axis=1, keepdims=True))
    A2 = exp_Z2 / np.sum(exp_Z2, axis=1, keepdims=True)
    return A1, A2

Exam questions

• [10 Marks] Explain why a single-layer perceptron cannot solve the XOR problem. Show how a Multi-Layer Perceptron (MLP) resolves this limitation.
• [8 Marks] Derive the Backpropagation weight update rule using the calculus chain rule for an artificial neural network with one hidden layer.