Lecture topic：Methodologies to Decompose End-to-End Backpropagation for Efficient Pipelined Training       

Speaker：Prof. Hung-Hsuan Chen 
Computer Science and Information Engineering, National Central University 

Time：2023.5.10 (Wed.) 15:30-17:20

Place :EC115(工程三館115 教室)

Host：Prof.Yu-Tai Ching (荊宇泰教授)

Abstract

Backpropagation (BP) is the cornerstone of today's deep learning algorithms. However, its inefficiency is partially caused by backward locking, where updating the weights of one layer locks the weight updates in other layers. This makes it challenging to update the weights in different layers simultaneously, limiting training efficiency. In this talk, I will present our recent research on methodologies to update parameters in different layers simultaneously through pipelining. Our design significantly improves the complexity of training time from O(nl) to O(n + l) (n is the number of training instances, and l is the number of hidden layers), while maintaining comparable testing accuracies to the models trained by BP.