Convolutional Neural Networks (CNNs) & Landmark Architectures

Explanation

Standard fully connected networks fail on high-resolution image data because parameter counts explode and spatial pixel locality is destroyed. CNNs exploit parameter sharing (the same filter detects edges across the entire image) and spatial hierarchy (early layers detect low-level edges; deeper layers detect complex textures and object parts). In very deep networks (>20 layers), training error degrades due to optimization difficulty; ResNet solves this by introducing residual skip connections: H(x) = F(x) + x, enabling gradients to flow unimpeded directly through identity mappings.

Key points

• Convolution Operation: Sliding learnable kernels/filters across 2D/3D spatial input to generate Feature Maps.
• Hyperparameters: Kernel Size (e.g., 3x3), Stride (step size), Padding ('valid' vs 'same').
• Output Dimension Formula: Output_Size = floor((Input_Size - Kernel_Size + 2*Padding) / Stride) + 1.
• Pooling Layers: Max Pooling and Average Pooling reduce spatial dimensions, providing translation invariance.
• Landmark Architectures: LeNet-5 (digits), AlexNet (ImageNet 2012, ReLU, Dropout), VGG-16 (small 3x3 filters), ResNet (Residual skip connections resolving degradation).

Example

Code examples

import torch
import torch.nn as nn

class SimpleCNN(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1)
        self.relu = nn.ReLU()
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)
        self.fc = nn.Linear(32 * 16 * 16, num_classes)

    def forward(self, x):
        x = self.pool(self.relu(self.conv1(x)))
        x = x.view(x.size(0), -1)
        return self.fc(x)

Exam questions

• [10 Marks] Explain the building blocks of a Convolutional Neural Network (Convolution, Activation, Pooling, Fully Connected layers). Calculate output dimensions given input parameters.
• [8 Marks] Explain the architectural innovation of ResNet (Residual Networks). How do skip connections overcome the degradation problem in deep networks?