Feedforward Neural Networks & Backpropagation Derivation

Explanation

Feedforward networks process inputs sequentially across layers without recurrent loops: a^[l] = g^[l](W^[l] a^[l-1] + b^[l]). During backpropagation, the partial derivative of the scalar loss L with respect to layer weights dL/dW^[l] is computed backward from the output layer. For a hidden layer, delta^[l] = (W^[l+1]^T delta^[l+1]) elementwise-multiplied by g'^l. The vanishing gradient problem arises when deep networks use Sigmoid or Tanh, whose derivatives saturate at values <= 0.25, exponentially diminishing gradients as they propagate backward through many layers.

Key points

• Universal Approximation Theorem: A feedforward network with a single hidden layer containing finite neurons and non-linear activations can approximate any continuous function.
• Activation Functions: Sigmoid sigma(z) = 1/(1+e^-z), Tanh tanh(z), ReLU max(0, z), Leaky ReLU max(0.01z, z), Softmax for multiclass probability.
• Cost Functions: Mean Squared Error (MSE) for regression; Cross-Entropy Loss for binary and multi-class classification.
• Backpropagation: Backward propagation of error using the Chain Rule of calculus to compute loss gradients with respect to every weight.

Example

Code examples

/*
Backpropagation Chain Rule Derivation:
For output layer L:
dLoss / dW^[L] = (dLoss / da^[L]) * (da^[L] / dz^[L]) * (dz^[L] / dW^[L])
Let delta^[L] = (a^[L] - y)  (for Cross-Entropy + Softmax)
Then:
dLoss / dW^[L] = delta^[L] * (a^[L-1])^T
dLoss / db^[L] = delta^[L]

For preceding hidden layer l:
delta^[l] = (W^[l+1]^T * delta^[l+1]) * g'^[l](z^[l])
dLoss / dW^[l] = delta^[l] * (a^[l-1])^T
*/

Exam questions

• [10 Marks] Derive the Backpropagation weight update equations step-by-step for a 3-layer neural network using the Chain Rule.
• [8 Marks] Explain the Vanishing Gradient problem in deep networks. Why does the Rectified Linear Unit (ReLU) activation function alleviate this issue?