Recurrent Neural Networks (RNNs), Vanishing Gradients & LSTMs
Idea in plain words
Recurrent Neural Networks (RNNs) process sequential temporal data by maintaining an internal hidden state: h_t = tanh(W_hh * h_{t-1} + W_xh * x_t + b_h).
Exam tip
High-yield exam topic (Very High).
Explanation
Feedforward networks assume all inputs and outputs are independent. Sequential tasks (machine translation, speech recognition, stock forecasting) depend inherently on historical context. Standard vanilla RNNs struggle with sequences longer than 10-20 steps because unrolling across time compounds matrix multiplication, causing gradients to vanish to zero. LSTMs (Hochreiter & Schmidhuber, 1997) counter this by creating an uninterrupted 'cell state highway' where information flows linearly with constant error carousels, modified only by additive gate interactions.
Key points
- Backpropagation Through Time (BPTT): Unrolls the network across time steps to compute temporal gradients.
- Vanishing and Exploding Gradients in RNNs: Repeated multiplication by weight matrices W_hh causes gradients to decay or explode exponentially over long sequences.
- Long Short-Term Memory (LSTM): Resolves long-term dependencies using an explicit Memory Cell (C_t) regulated by three multiplicative gates.
- LSTM Gates: Forget Gate (f_t: what to discard), Input Gate (i_t: what new info to store), Output Gate (o_t: what to output based on cell state).
- Gated Recurrent Unit (GRU): Compact alternative merging cell state and hidden state, utilizing Reset and Update gates.
Example
Example
In natural language processing, when predicting the verb in 'The cats that lived in the barn ... WERE hungry', the forget and input gates preserve the plural context 'cats' across long intervening clauses.
Code examples
/*
LSTM Cell Equations at time step t:
1. Forget Gate:
f_t = sigma(W_f * [h_{t-1}, x_t] + b_f)
2. Input Gate & Candidate State:
i_t = sigma(W_i * [h_{t-1}, x_t] + b_i)
C_tilde_t = tanh(W_c * [h_{t-1}, x_t] + b_c)
3. Cell State Update:
C_t = f_t * C_{t-1} + i_t * C_tilde_t
4. Output Gate & Hidden State:
o_t = sigma(W_o * [h_{t-1}, x_t] + b_o)
h_t = o_t * tanh(C_t)
*/
Exam questions
- [10 Marks] Explain the architecture and mathematical equations of an LSTM cell. Detail the purpose of the Forget Gate, Input Gate, and Output Gate.
- [8 Marks] Explain Backpropagation Through Time (BPTT) and analyze why vanilla RNNs fail on long-term sequential dependencies.