New optimization algorithms for neural network training using operator splitting techniques

Abstract

We present a new type of optimization algorithms, adapted for neural network training. These algorithms are based upon sequential operator splitting technique for some associated dynamical systems.

Furthermore, we investigate through numerical simulations the empirical rate of convergence of these iterative schemes toward a local minimum of the loss function, with some suitable choices of the underlying hyper parameters.

We validate the convergence of these optimizers using the results of the accuracy and of the loss function on the MNIST, MNIST-Fashion and CIFAR 10 classification datasets.

Authors

C.D. Alecsa
Faculty of Mathematics and Computer Science, Babes-Bolyai University, Cluj-Napoca, Romania
Tiberiu Popoviciu Institute of Numerical Analysis (Romanian Academy), Cluj-Napoca, Romania

T. Pinta
Mathematical Institute University of Oxford, Oxford, England

I. Boros
Faculty of Mathematics and Computer Science, Babes-Bolyai University, Cluj-Napoca, Romania
Tiberiu Popoviciu Institute of Numerical Analysis (Romanian Academy), Cluj-Napoca, Romania

Keywords

unconstrained optimization problems; splitting; neural network training;

Paper coordinates

C.-D. Alecsa, T. Pinta, I. Boros, New optimization algorithms for neural network training using operator splitting techniques, 126 (2020), pp. 178-190, doi: 10.1016/j.neunet.2020.03.018

PDF

About this paper

Publisher Name

Elsevier

Print ISSN

0893-6080

Online ISSN

google scholar link

Related Posts

Menu