a Caputo fractional-order gradient descent for neural network training¶
Implements a Caputo fractional-order gradient descent for neural network training, with an optimal per-dataset fractional order and Firefly-based initialization.
This work replaces the integer-order gradient in backpropagation with a Caputo fractional derivative of the loss, so each weight update carries the memory and non-locality characteristic of fractional calculus, which the authors argue yields a more gradual and controlled adaptation to the data. Fractional backpropagation is derived for both the cross-entropy and mean-square-error losses, the resulting fractional gradient descent is shown to converge linearly, and the fractional order is treated as a tunable hyperparameter whose optimal value is selected per dataset. A hybrid scheme uses the Firefly metaheuristic optimizer to initialize the weights before fractional gradient descent refines them.
The method generalizes gradient descent to a Caputo fractional gradient, studied through convergence analysis and empirical evaluation on neural-network training.
Reference: Priyanka Harjule, Rinki Sharma, Rajesh Kumar, "Fractional-order gradient approach for optimizing neural networks: A theoretical and empirical analysis", Chaos, Solitons & Fractals 2025. https://doi.org/10.1016/j.chaos.2025.116009