Generative Models: Autoencoders, VAEs & Generative Adversarial Networks
Idea in plain words
Discriminative Models (p(y|x)) classify inputs; Generative Models (p(x) or p(x,y)) learn underlying data distributions to generate novel synthetic samples.
Exam tip
High-yield exam topic (High).
Explanation
Generative modeling seeks to sample new data points from high-dimensional probability distributions. Standard autoencoders compress data but their latent spaces have arbitrary holes, preventing smooth random sampling. VAEs enforce a continuous, probabilistic latent prior by optimizing the Evidence Lower Bound (ELBO), balancing reconstruction loss with KL divergence. GANs (Goodfellow, 2014) abandon explicit density estimation, framing training as a zero-sum game: min_G max_D V(D, G) = E_{x}[log D(x)] + E_{z}[log(1 - D(G(z)))]. At Nash equilibrium, the Generator synthesizes outputs indistinguishable from real data (D(x) = 0.5).
Key points
- Autoencoders (AE): Unsupervised networks consisting of an Encoder (compresses input into low-dimensional bottleneck z) and Decoder (reconstructs input from z).
- Variational Autoencoders (VAE): Probabilistic generative models enforcing that the latent space follows a standard normal Gaussian distribution N(0, I) via Kullback-Leibler (KL) divergence loss.
- Reparameterization Trick: Enables backpropagation through stochastic nodes: z = mu + sigma * epsilon, where epsilon ~ N(0, I).
- Generative Adversarial Networks (GANs): Minimax game between a Generator G (creates realistic fakes) and Discriminator D (distinguishes real from fake).
Example
Example
Deepfake image generation and super-resolution upscaling synthesized by Deep Convolutional GANs (DCGANs).
Code examples
/*
GAN Minimax Game Formulation:
min_G max_D V(D, G) = E_{x ~ p_data(x)} [log D(x)] + E_{z ~ p_z(z)} [log(1 - D(G(z)))]
Training Loop:
1. Update Discriminator D:
Maximize log D(x_real) + log(1 - D(G(z_noise)))
2. Update Generator G:
Minimize log(1 - D(G(z_noise))) [or Maximize log D(G(z_noise))]
*/
Exam questions
- [10 Marks] Explain the minimax game architecture of Generative Adversarial Networks (GANs). Write the objective function and explain the roles of Generator and Discriminator.
- [8 Marks] Describe Variational Autoencoders (VAEs). Explain the Reparameterization Trick and why it is necessary for gradient backpropagation.