Adaptive Parameter Fractional-Order Gradient Descent Learning¶

Implements Adaptive Parameter Fractional-Order Gradient Descent Learning (AP-FOGDL), a Caputo-derivative gradient descent with a convergence-guided adaptive learning rate.

Classical fractional-order gradient descent replaces the integer gradient \(\nabla f(\theta_t)\) with the Caputo fractional gradient \({}^C_c D^{\alpha}_{\theta} f(\theta_t)\), which carries memory of past iterates through the fractional order \(\alpha\). Taking the lower terminal \(c\) to be the previous iterate and approximating the Caputo derivative by a truncated Taylor expansion yields a usable update; a positive offset \(\delta\) keeps the fractional term from stalling when consecutive iterates coincide. AP-FOGDL builds on this form by replacing the fixed step with an adaptive learning rate \(\eta_t\) derived from computable upper bounds on the iterates, which the authors use to establish convergence without assuming a priori boundedness of the weights, and instantiates the same scheme for the Riemann-Liouville derivative as well.

\[ \begin{aligned} \theta_{t+1} &= \theta_t - \eta_t \, \frac{\nabla f(\theta_t)}{\Gamma(2-\alpha)} \, \bigl(\lvert \theta_t - \theta_{t-1}\rvert + \delta\bigr)^{1-\alpha}, \\ {}^C_c D^{\alpha}_{\theta} f(\theta) &= \frac{1}{\Gamma(n-\alpha)} \int_{c}^{\theta} \frac{f^{(n)}(\tau)}{(\theta-\tau)^{\alpha-n+1}} \, d\tau, \qquad n-1 < \alpha < n. \end{aligned} \]

where \(\theta\) are the parameters, \(\eta_t\) the adaptive learning rate, \(\nabla f(\theta_t)\) the gradient, \(\alpha \in (0,1)\) the fractional order controlling the memory effect, \(\delta>0\) a stabilizing offset, \(\Gamma(\cdot)\) the gamma function, and \(c\) the lower terminal of the Caputo derivative (taken as the previous iterate).

Reference: Mingjie Ma, Siyuan Chen, Lunan Zheng, "Novel adaptive parameter fractional-order gradient descent learning for stock selection decision support systems", European Journal of Operational Research 2025. https://doi.org/10.1016/j.ejor.2025.01.013

Back to the Canon