Stochastic inertial projection-extrapolation method: A damped approach with applications to machine learning

We introduce and analyze a novel stochastic algorithm for monotone variational inequalities under uncertainty, a class that includes convex–concave saddle-point problems and convex optimization problems as special cases. We propose the Stochastic Inertial Projection-Extrapolation Method (SIPEM), which combines three powerful techniques: damped inertial momentum (−1<θ≤0), proximal steps for non-smooth regularizers, and a stochastic extra-gradient step that reuses past gradient information for variance reduction. Unlike classical inertial methods that employ positive momentum for acceleration, our damped inertial momentum serves as a smoothing mechanism that stabilizes iterates against stochastic noise. The algorithm operates under a standard unbiasedness assumption and a realistic bounded variance condition on the stochastic oracle, weaker and more widely applicable than the common bounded (sub)gradient assumption. We provide a rigorous non-asymptotic convergence analysis, establishing an upper bound on the expected restricted gap function of the averaged iterates. The method with constant step sizes, converges at a rate of O(1/N), matching the optimal complexity of stochastic extra-gradient methods. Extensive numerical experiments on sparse logistic regression, distributionally robust optimization, and training of generative adversarial networks (GANs) on the MNIST dataset demonstrate that SIPEM improves upon Stochastic Double Inertia Forward-Backward-Forward (DIFBF), Stochastic Extragradient (SEG), and Stochastic Forward-Backward Momentum (SFBM) in terms of computation time, number of iterations, and stability, showing particular promise for adversarial learning tasks where oscillations and mode collapse are critical issues.