Lectures

You can download the lectures here. We will try to upload lectures prior to their corresponding classes.

• (dl-00) Introduction
  tl;dr:
  [slides]
  

  Suggested Readings:

  • Chapter 1 of the AI textbook by Russell and Norvig

  Additional References

  • Know your brain
  • McCulloch-Pitts Neuron, 1943
  • Hebbian Learning Princile by D. Hebb, 1949
  • Neocognitron by Fukushima, 1980 and an excellent Video Demo
  • Backprop for training NNs by Rumelhart et al. 1986
  • Backprop applied to MNIST by Lecun, 1989
  • AlexNet by Krizhevsky et al. 2012
  • ResNet by He et al. 2015
• (dl-01) Artificial Neuron Models (MP neuron and Perceptron)
  tl;dr: The Basic building block of ANNs.
  [perceptron-learning-code] [slides]
  

  Suggested Readings:

  • Chapter-1 of Michael Nielson’s NNDL textbook
  • Convergence proof for Perceptron
  • MP Neuron
• (dl-02) Network of perceptrons
  tl;dr: If not one, can multiple (network) of Perceptrons do the job?.
  [slides]
  

  Suggested Readings:

  • Chapter-1 of Michael Nielson’s NNDL textbook
• (dl-03) Representational Power of an MLP
  tl;dr: Multi Layered Network of Neurons can represent any artbirary function!
  [ReLU-activation-code] [sigmoid-activation-codes] [slides]
  

  Suggested Readings:

  • Chapter 4 of Michael Nielson’s NNDL book
  • Deep Mind Blog on Universal Approximation Theorem
  • Approximation by Superposition of Sigmoid Functions
  • Approximation Capabilities of Muitilayer Feedforward Networks
• (dl-04) Gradient Descent
  tl;dr: Its all about finding the right set of model parameters!
  [slides]
  

  Suggested Readings:

  • ML chapter from Goodfellow et al. DL Book
  • Numerical Computation chapter from Goodfellow et al DL book
• (dl-05) Backpropagation-1
  tl;dr: Gradient Descent applied to a simple computational graph
  [slides]
  

  Suggested Reading

  • Chapter 2 in Michael Nielson’s NNDL textbook

  Additional Readings:

  • Paper by Rumelhart et al. that popularized backpropagation
• (dl-06) Backpropagation-2
  tl;dr: Gradient Descent applied to a generic computational graph (DAG)
  [slides]
  

  Suggested Reading

  • Chapter 2 in Michael Nielson’s NNDL textbook

  Additional Readings:

  • Paper by Rumelhart et al. that popularized backpropagation
• (dl-07) Cross-Entropy Loss
  tl;dr: The loss function for training the NN classifiers
  [slides]
  

  Suggested Readings:

  • Chap-3 from Goodfellow et al. DL 2016 book
  • Chap-3 from Michael Nielson’s NNDL book
• (dl-08) Training DNNs - I
  tl;dr: Issues with Gradient Descent, specifically with the learning rate
  [codes - Optimizing 1D Convex function] [codes - Optimizing 2D Quadratic function] [slides]
  

  Suggested Readings:

  • Ruder’s blog on variants of gradient-based updates
  • Chapter on Optimization from Goodfellow et al. book
  • Chapter 5 from Michael Nielson’s NNDL book
• (dl-09) Training DNNs - II
  tl;dr: Normalization and Regularization.
  [slides]
  

  Suggested Readings:

  • Chapter on Regularization from Goodfellow et al. book
  • Chapter 5 from Michael Nielson’s NNDL book
• (dl-10) Building Blocks of CNNs
  tl;dr: Basic constituting elements of Convolutional Neural Networks (CNN)
  [slides] [1D-conv-backprop] [2D-conv-backprop]
  

  Suggested Readings:

  • CNNs module from CS231n
  • CNNs Chapter from Goodfellow et al. DL book
• (dl-11) Evolution of CNN Architectures
  tl;dr: How different families of architectures evolved over time.
  [slides]
  

  Suggested Readings:

  • CNNs module from CS231n
  • CNNs Chapter from Goodfellow et al. DL book
• (dl-12) Recurrent Neural Networks
  tl;dr: Deep Learning Models to handle Sequence prediction tasks
  [slides]
  

  Suggested Readings:

  • Chapter 10 from Goodfellow et al DL book
  • Denny’s tutorial on RNNs
  • Colah’s blog on LSTMs
  • GRU by Cho et al. 2014
• (dl-13) Word Embeddings
  tl;dr: Representing the words as vectors for learning to perfrom language tasks
  [slides]
  

  Suggested Readings:

  • Notes by Elena Voita
  • GloVe Paper
  • Word2Vec Paper
• (dl-14) Encoder-Decoder Models and Attention
  tl;dr: Configuration of a NN pair towards sequence learning problems and learning to selectively focus
  [slides]
  

  Suggested Readings:

  • Lecture Notes/Material on by Elena Voita
  • Sequence to sequence learning by Sutskever et al. NeurIPS 2014
  • Neural Machine Translation with aligning by Bahdanau et al. ICLR 2015
  • Show Attend and Tell by Xu et al. 2015
• (dl-15) Self-Attention and Transformers
  tl;dr: Attention is all we need!
  [slides-I] [slides-II] [GIF-1 (decoding)] [GIF-2 (decoding)]
  

  Suggested Readings:

  • Chapter-12 from the Bishop’s book
  • The Illustrated Transformer
  • Attention is all we need!
  • Computational complexity discussion
  • My slides on self-attention layer for CNNs
• (dl-16) Applications of Transformers
  tl;dr: Different configurations of Transformers that realize a variety of applications.
  [slides]
  

  Suggested Readings:

  • Chapter-12 from the Bishop’s book
• (dl-17) Autoencoders
  tl;dr: Models that map input to itself!
  [slides]
  

  Suggested Readings:

  • Chapter-19 from the Bishop’s book
  • Chapter-14 from Goodfellow et. al.’s book
• (dl-18) Generative Models
  tl;dr: Tools to model the density over the space of data.
  [slides]
  

  Suggested Readings:

  • IJCAI-ECAI 2018 Tutorial on Generative Modeling
• (dl-19) Variational Autoencoders
  tl;dr: Autoencoders but also generative models!
  [slides]
  

  Suggested Readings:

  • Joseph Rocca’s blog
• (dl-20) Generative Adversarial Networks (GAN)
  tl;dr: Pair of NNs that compete adversarially to realize an implicit density modelling!
  [slides]
  

  Suggested Readings:

  • Dive into DL, chapter on GANs
  • DC GAN by Radford et al. NeurIPS 2016
• (dl-21) Diffusion Models
  tl;dr: Process of successively destroying and recovering the samples
  [slides]
  

  Suggested Readings:

  • Understanding Diffusion Models: A Unified Perspective
  • Lilian Weng’s blog