Lectures

You can download the lectures here. We will try to upload lectures prior to their corresponding classes.

• (dl-0) Introduction
  tl;dr: Introduction to Deep Learning, brief history of Artificial Neural Networks and logistics of this course.
  [slides]
  

  Suggested Readings:

  • McCulloch-Pitts Neuron, 1943
  • Hebbian Learning Princile by D. Hebb, 1949
  • Neocognitron by Fukushima, 1980
  • Backprop for training NNs by Rumelhart et al. 1986
  • Backprop applied to MNIST by Lecun, 1989
  • AlexNet by Krizhevsky et al. 2012
  • ResNet by He et al. 2015
• (dl-1) Artificial Neuron Models (MP neuron and Perceptron)
  tl;dr: The Basic building block of ANNs.
  [codes] [slides]
  

  Suggested Readings:

  • Convergence proof for Perceptron
  • MP Neuron
• (dl-2) Shortcomings of Perceptron
  tl;dr: Perceptron is not perfect!
  [slides]
  

  Suggested Readings:

• (dl-3) MLP - Representational Power of Networks of Neurons
  tl;dr: Multi Layered Network of Neurons can learn any artbirary function!
  [slides]
  

  Suggested Readings:

  • Approximation by Superposition of Sigmoid Functions
  • Approximation Capabilities of Muitilayer Feedforward Networks
  • Chapter 4 of Michael Nielson’s book
• (dl-4) Optimization (Gradient Descent)
  tl;dr: Its all about finding the right set of model parameters!
  [slides]
  

  Suggested Readings:

• (dl-5) Backpropagation-1
  tl;dr: Gradient Descent applied to a simple computational graph
  [slides]
  

  Suggested Readings:

  • Paper by Rumelhart et al. that popularized backpropagation
• (dl-6) Backpropagation-2
  tl;dr: Gradient Descent applied to an MLP and more!
  [slides]
  

  Suggested Readings:

  • Chap-2 from Michael Nielson’s online book
  • Forward Forward Algorithm by Prof. Hinton
• (dl-7) Cross-Entropy Loss
  tl;dr: The loss function for training the NN classifiers
  [slides]
  

  Suggested Readings:

  • Chap-3 from Goodfellow et al. 2016 book
• (dl-8) Building Blocks of CNNs
  tl;dr: Basic constituting elements of Convolutional Neural Networks (CNN)
  [slides]
  

  Suggested Readings:

• (dl-9) Evolution of CNN Architectures
  tl;dr: How different families of architectures evolved over time
  [slides]
  

  Suggested Readings:

• (dl-10) Training DNNs - 1
  tl;dr: Important aspects of training DNNs
  [slides]
  

  Suggested Readings:

  • Chap-7 on Regularization from I Goodfellow et al. 2016
  • Chap-8 on Optimization for DNNs from I Goodfellow et al. 2016
• (dl-11) Training DNNs - II
  tl;dr: Important aspects of training DNNs - II
  [slides]
  

  Suggested Readings:

  • Chap-7 on Regularization from I Goodfellow et al. 2016
  • Chap-8 on Optimization for DNNs from I Goodfellow et al. 2016
• (dl-12) Visualizing and Understanding CNNs
  tl;dr: What do the CNNs learn? Why they predict what they predict?
  [slides]
  

  Suggested Readings:

  • Rich Feature Hierarchies by Girshick et al. CVPR 2014
  • Visualizing and Understanding CNNs by Zeiler and Fergus ECCV 2014
  • Deep Inside CNNs by Simonyan et al. ICLRW 2014
  • Class Activation Maps (CAM) by B Zhou et al. CVPR 2016
  • Grad-CAM by Selvaraju et al. NIPSW 2016, ICCV 2017
  • CNN-Fixations by Mopuri et al. TIP 2018
  • Texture Synthesis using CNNs by Gatys et al. NeurIPS 2015
  • DeepDream
  • Neural style transfer by Gatys et al. CVPR 2016
  • Understanding Deep Representations by Inverting them by A Mahendran et al. CVPR 2015
  • Inverting Visual Representations with CNNs by A Dosovitskiy et al. CVPR 2016-
• (dl-13) Recurrent Neural Networks
  tl;dr: Deep Learning Models to handle Sequence prediction tasks
  [slides]
  

  Suggested Readings:

  • Denny’s tutorial on RNNs
  • Colah’s blog on LSTMs
  • GRU by Cho et al. 2014
• (dl-14) Word Embeddings
  tl;dr: How do we represent words (or, language) as vectors?
  [slides]
  

  Suggested Readings:

• (dl-15) Encoder-Decoder Models and Attention
  tl;dr: Configuration of a NN pair towards sequence learning problems and learning to selective focus
  [slides]
  

  Suggested Readings:

  • Sequence to sequence learning by Sutskever et al. NeurIPS 2014
  • Neural Machine Translation with aligning by Bahdanau et al. ICLR 2015
  • Show Attend and Tell by Xu et al. 2015
• (dl-16) Self-Attention and Transformers
  tl;dr: Attention is all we need!
  [slides]
  

  Suggested Readings:

  • The Illustrated Transformer
  • Attention is all we need!
  • Computational complexity discussion
  • My slides on self-attention layer for CNNs
• (dl-17) Generative Models
  tl;dr: Tools to model the density over the space of data.
  [slides]
  

  Suggested Readings:

  • IJCAI-ECAI 2018 Tutorial on Generative Modeling
• (dl-18) Autoencoders
  tl;dr: Models that map input to itself!
  [slides]
  

  Suggested Readings:

• (dl-19) Variational Autoencoders
  tl;dr: Autoencoders but also generative models!
  [slides]
  

  Suggested Readings:

• (dl-20) Generative Adversarial Networks (GAN)
  tl;dr: Pair of NNs that compete adversarially to realize an implicit density modelling!
  [slides]
  

  Suggested Readings:

  • Dive into DL, chapter on GANs
  • DC GAN by Radford et al. NeurIPS 2016