FOGDM-RBF¶

Implements FOGDM-RBF, fractional-order gradient descent with momentum for training RBF neural networks.

Classical gradient descent uses the integer-order gradient, which depends only on the present iterate. FOGDM-RBF instead descends along a Caputo fractional-order gradient: replacing \(\nabla E\) by its order-\(\alpha\) Caputo derivative injects a memory of the path toward the current weight, controlled by the non-integer order \(\alpha \in (0,1)\). Taking the lower terminal of the Caputo operator at the previous iterate gives the closed form below.

A momentum term is then layered on top of this fractional gradient to damp the oscillations that plague RBF training, and the step is driven by an adaptive learning rate \(\eta\) to accelerate convergence.

\[ \begin{aligned} \tilde{g}_t &= \frac{1}{\Gamma(2-\alpha)}\, g_t \, \lvert \theta_t - \theta_{t-1} \rvert^{\,1-\alpha} \\ \Delta\theta_t &= \mu\,\Delta\theta_{t-1} - \eta\,\tilde{g}_t \\ \theta_{t+1} &= \theta_t + \Delta\theta_t \end{aligned} \]

where \(\theta\) are the network weights, \(g_t = \nabla E(\theta_t)\) is the ordinary gradient of the error \(E\), \(\tilde{g}_t\) is the Caputo fractional-order gradient of order \(\alpha \in (0,1)\), \(\Gamma\) is the gamma function, \(\eta\) is the (adaptive) learning rate, and \(\mu\) is the momentum coefficient. As \(\alpha \to 1\) the fractional gradient reduces to \(g_t\) and the rule recovers standard gradient descent with momentum.

Reference: Han Xue, "Fractional-order gradient descent with momentum for RBF neural network-based AIS trajectory restoration", Soft Computing 25, 869-882, 2021. https://doi.org/10.1007/s00500-020-05484-5

Back to the Canon