This study aims to develop a stochastic optimization algorithm that accelerates and stabilizes convergence in large-scale nonconvex machine learning problems. Motivated by the limitations of stochastic gradient descent and existing stochastic conjugate gradient methods, we propose an inertial-accelerated framework that incorporates both momentum and correction strategies to enhance convergence efficiency and robustness.
The proposed Projected Stochastic Accelerated Three-Term Conjugate Gradient (PSATCG) algorithm extends the three-term conjugate gradient framework by introducing a two-step inertial acceleration and a modified correction term. An improved inexact line search strategy ensures global convergence under mild conditions. Theoretical analysis establishes linear convergence, and numerical experiments are conducted on two nonconvex machine learning models across nine benchmark datasets.
Experimental results demonstrate that PSATCG consistently achieves faster and more stable convergence than stochastic gradient descent (SGD), SAGA, SARAH and adaptive optimizers such as Adam and RMSprop. The algorithm maintains strong robustness under weak regularization and noisy conditions, validating the theoretical findings on stability and linear convergence.
This work introduces two major innovations: a two-step inertial acceleration mechanism that leverages multi-iterative momentum to enhance convergence, and a correction term that stabilizes search directions in nonconvex landscapes. Together, these techniques establish a new class of stochastic conjugate gradient methods with stronger theoretical guarantees and improved empirical performance for large-scale machine learning optimization.
