Data-free Knowledge Extraction from Deep Neural Networks

Welcome to the tutorial on Data-free Knowledge Extraction to be held as part of the NCVPRIPG 2023 conference.


Data-free Knowledge Extraction (DFKE) refers to extracting useful information from a trained deep neural network (DNN) without accessing the underlying training data over which the DNN is trained. The extracted information can be diverse. For instance, it can be a replica of the DNN itself, some sensitive information about the underlying training data, patterns from thereof, etc. DFKE can be extremely vexing, particularly in deployments like MLaaS (Machine Learning as a service). Considering the amount of data, human expertise, and computational resources typically required to learn sophisticated DNNs, it is natural to consider them intellectual property. Therefore, they need to be protected against any such attempts of extraction (referred to as attacks). On the other hand, philosophically, it would be interesting to (i) understand the utility of these trained models without their training data and (ii) formulate guarantees on the information leakage (or extraction). In this tutorial, I plan first to introduce the phenomenon of data-free model extraction and discuss different ways in which it can be manifested, both in white-box and black-box scenarios. Later, I will focus more on the potential threats of leaking sensitive information about the training data to a dishonest user in the form of different attacks. Finally, I will discuss some of the active directions to investigate further.

Topics to be discussed (may not be in the same sequence)
  • Introduction
    • Deploying Deep models post-training: What and What not?
    • Knowledge Distillation
    • Noise Optimization towards CNN visualization
    • Generative Adversarial Networks (GAN)
  • Data-free Knowledge Distillation (towards creating a replica of the target model)
    • Via Noise Optimization
    • Via Generative Reconstruction
    • Adversarial Exploration
  • Data-free attacks (towards extracting sensitive information about the training data)
  • Conclusion and Future Directions
Slides of the tutorial
  • Session 1: 2 - 4 PM, July 21, 2023 (Friday)
  • Session 2: 4.30 - 5.30 PM, July 21, 2023 (Friday)
Bibliography and References