Deep Network Optimization, Weight Initialization & Regularization

Explanation

Training deep networks requires navigating complex non-convex loss surfaces filled with saddle points and ravines. Standard SGD oscillates across narrow ravines; Momentum incorporates exponentially decaying moving averages of past gradients to accelerate along relevant dimensions. Adam (Adaptive Moment Estimation) computes adaptive learning rates per parameter, tracking both the first moment (mean) and second raw moment (uncentered variance). Batch Normalization reduces internal covariate shift, allowing higher learning rates and reducing dependence on delicate weight initialization.

Key points

• Advanced Optimizers: Momentum (accelerates in consistent directions), RMSprop (scales by moving average of squared gradients), Adam (combines Momentum + RMSprop).
• Weight Initialization: Zero initialization causes symmetry failure; Xavier/Glorot initialization for Tanh; He/Kaiming initialization for ReLU.
• Batch Normalization: Normalizes hidden layer inputs across mini-batch (zero mean, unit variance) to stabilize internal covariate shift.
• Regularization in Deep Learning: Dropout (randomly deactivates neurons during training with probability p), L2 Weight Decay, Early Stopping.

Example

Code examples

/*
Adam Optimizer Algorithm:
m_t = beta1 * m_{t-1} + (1 - beta1) * g_t       (First moment: Momentum)
v_t = beta2 * v_{t-1} + (1 - beta2) * (g_t)^2   (Second moment: RMSprop)

Bias corrections:
m_hat = m_t / (1 - beta1^t)
v_hat = v_t / (1 - beta2^t)

Weight update:
theta_t = theta_{t-1} - (alpha / (sqrt(v_hat) + epsilon)) * m_hat
(Standard defaults: beta1=0.9, beta2=0.999, epsilon=1e-8)
*/

Exam questions

• [10 Marks] Explain the Adam optimization algorithm. How does it combine the advantages of Momentum and RMSprop?
• [8 Marks] Discuss Dropout and Batch Normalization. Explain how each prevents overfitting and stabilizes deep network training.