Lectures

You can download the lectures here. We will try to upload lectures prior to their corresponding classes.

• (dl-00) Introduction
  tl;dr:
  [slides]
  

  Suggested Readings:

  • Chapter 1 of the AI textbook by Russell and Norvig

  Additional References

  • Know your brain
  • McCulloch-Pitts Neuron, 1943
  • Hebbian Learning Princile by D. Hebb, 1949
  • Neocognitron by Fukushima, 1980 and an excellent Video Demo
  • Backprop for training NNs by Rumelhart et al. 1986
  • Backprop applied to MNIST by Lecun, 1989
  • AlexNet by Krizhevsky et al. 2012
  • ResNet by He et al. 2015
• (dl-01) Artificial Neuron Models (MP neuron and Perceptron)
  tl;dr: The Basic building block of ANNs.
  [perceptron-learning-code] [slides]
  

  Suggested Readings:

  • Chapter-1 of Michael Nielson’s NNDL textbook
  • Convergence proof for Perceptron
  • MP Neuron
• (dl-02) Network of perceptrons
  tl;dr: If not one, can multiple (network) of Perceptrons do the job?.
  [slides]
  

  Suggested Readings:

  • Chapter-1 of Michael Nielson’s NNDL textbook
  • Chapter-4 and 5 of Bishops’ book
• (dl-03) Representational Power of an MLP
  tl;dr: Multi Layered Network of Neurons can represent any artbirary function!
  [ReLU-activation-code] [sigmoid-activation-codes] [slides]
  

  Suggested Readings:

  • Chapter 4 of Michael Nielson’s NNDL book
  • Deep Mind Blog on Universal Approximation Theorem
  • Approximation by Superposition of Sigmoid Functions
  • Approximation Capabilities of Muitilayer Feedforward Networks
• (dl-04) Gradient Descent
  tl;dr: Its all about finding the right model parameters!
  [slides]
  

  Suggested Readings:

  • ML chapter from Goodfellow et al. DL Book
  • Numerical Computation chapter from Goodfellow et al DL book
• (dl-05) Backpropagation-1
  tl;dr: Gradient Descent applied to a simple computational graph
  [slides]
  

  Suggested Reading

  • Chapter 2 in Michael Nielson’s NNDL textbook

  Additional Readings:

  • Paper by Rumelhart et al. that popularized backpropagation
• (dl-06) Backpropagation-2
  tl;dr: Gradient Descent applied to a generic computational graph (DAG)
  [slides]
  

  Suggested Reading

  • Chapter 2 in Michael Nielson’s NNDL textbook

  Additional Readings:

  • Paper by Rumelhart et al. that popularized backpropagation
• (dl-08) Training DNNs - I
  tl;dr: Issues with Gradient Descent, specifically with the learning rate
  [codes - Optimizing 1D Convex function] [codes - Optimizing 2D Quadratic function] [slides]
  

  Suggested Readings:

  • Chapter-7 from Bishop’s Book
  • Ruder’s blog on variants of gradient-based updates
  • Chapter on Optimization from Goodfellow et al. book
  • Chapter 5 from Michael Nielson’s NNDL book
• (dl-09) Training DNNs - II
  tl;dr: Normalization and Regularization.
  [slides]
  

  Suggested Readings:

  • Chapter 9 on Regularization from Bishop’s book
  • Chapter on Regularization from Goodfellow et al. book
  • Chapter 5 from Michael Nielson’s NNDL book
• (dl-10) Building Blocks of CNNs
  tl;dr: Basic constituting elements of Convolutional Neural Networks (CNN)
  [slides] [1D-conv-backprop] [2D-conv-backprop]
  

  Suggested Readings:

  • CNNs module from CS231n
  • Chapter 10 from Bishop’s book
  • CNNs Chapter from Goodfellow et al. DL book
• (dl-11) Evolution of CNN Architectures
  tl;dr: How different families of architectures evolved over time.
  [slides]
  

  Suggested Readings:

  • CNNs module from CS231n
  • CNNs Chapter from Goodfellow et al. DL book
  • CNNs Chapter from Bishop’s book
• (dl-11a) Visualizing and Understanding CNNs
  tl;dr: What do the CNNs learn? Why the predict what they predict?
  [slides]
  

  Suggested Readings:

  • Rich Feature Hierarchies by Girshick et al. CVPR 2014
  • Visualizing and Understanding CNNs by Zeiler and Fergus ECCV 2014
  • Deep Inside CNNs by Simonyan et al. ICLRW 2014
  • Class Activation Maps (CAM) by B Zhou et al. CVPR 2016
  • Grad-CAM by Selvaraju et al. NIPSW 2016, ICCV 2017
  • CNN-Fixations by Mopuri et al. TIP 2018
  • Texture Synthesis using CNNs by Gatys et al. NeurIPS 2015
  • DeepDream
  • Neural style transfer by Gatys et al. CVPR 2016
  • Understanding Deep Representations by Inverting them by A Mahendran et al. CVPR 2015
  • Inverting Visual Representations with CNNs by A Dosovitskiy et al. CVPR 2016-
• (dl-12) Recurrent Neural Networks
  tl;dr: Deep Learning Models to handle Sequence prediction tasks
  [slides]
  

  Suggested Readings:

  • Chapter 10 from Goodfellow et al DL book
  • Denny’s tutorial on RNNs
  • Colah’s blog on LSTMs
  • GRU by Cho et al. 2014
• (dl-13) Word Embeddings
  tl;dr: Representing the words as vectors for learning to perfrom language tasks
  [slides]
  

  Suggested Readings:

  • Notes by Elena Voita
  • GloVe Paper
  • Word2Vec Paper