The Improved Adaptive Algorithm of Deep Learning with Barzilai-Borwein Step Size

PubDate: Aug 2022

Teams:  Wenzhou University；Hangzhou Normal University

Writers: Zhi-Jun Wang; He-Bei Gao; Bin-Shuang Zhang

PDF: The Improved Adaptive Algorithm of Deep Learning with Barzilai-Borwein Step Size

Abstract

To solve the problem that it is difficult to determine the learning rate when training a neural network model, this paper proposes an improved adaptive algorithm based on the Barzilai-Borwein (BB) step size. In this paper, the new algorithm accelerates the model's training through the second-order momentum and adapts the learning rate according to the BB step size. We also set an adequate range for the learning rate to ensure the stability of adaptive adjustment and reduce the error of step size. Compared with different algorithms in a series of popular models, the new algorithm significantly avoids the tediousness of manually adjusting the learning rate and helps to improve the convergence speed. The results show that the new algorithm is feasible and effective.