Toward Generating Efficient Deep Neural Networks

Recent advances in deep neural networks have led to tremendous applications in various tasks, such as object classification and detection, image synthesis, natural language processing, game playing, and biological imaging. However, deploying these pre-trained networks on resource-limited devices poses a challenge, as most state-of- the-art networks contain millions of parameters, making them cumbersome and slow in real-world applications. To address this problem, numerous network compression and acceleration approaches, also known as efficient deep neural networks or efficient deep learning, have been investigated, in terms of hardware and software (algorithms), training, and inference. The aim of this dissertation is to study several algorithms, with a particular focus on network pruning and knowledge distillation, which have been identified as powerful techniques in enabling efficient processing without significant performance degradation. While these general methods are not limited to certain network structures, datasets, and tasks, we focus on compressing and accelerating popular convolutional neural networks (CNNs) for image classifications, for which these methods were originally developed and conventionally evaluated with extensive empirical study. In network pruning, we explore an important yet largely neglected aspect of network pruning, the efficiency of the pruning procedure itself. Our contributions to network pruning focus on post-training channel pruning, which is usually com- putationally intensive and heavily energy-consuming. A typical pruning procedure consists of iterative procedures of ranking, pruning, and fine-tuning. We challenge the common belief of the importance of ranking criteria with empirical studies and propose an efficient pipeline for pruning CNNs by integrating ranking and fine-tuning through computational re-usage. We evaluate the proposed method with extensive experimental studies. Our research in knowledge distillation, more precisely online knowledge distilla- tion, is motivated by the observation that training networks using existing online KD approaches is a highly dynamic procedure, in which each student network learns its parameters from scratch and acts as an instructor for other networks. Naturally, training with developing instructors tends to involve more uncertainty and fluctuation. To generate superior and robust knowledge, we focus on leveraging various information encoded in each peer’s learning trajectory to dynamically construct superior teachers to supervise other students, potentially improving the performance of students during inference.