Controlled Descent Training
Journal article, 2024

In this work, a novel and model-based artificial neural network (ANN) training method is developed supported by optimal control theory. The method augments training labels in order to robustly guarantee training loss convergence and improve training convergence rate. Dynamic label augmentation is proposed within the framework of gradient descent training where the convergence of training loss is controlled. First, we capture the training behavior with the help of empirical Neural Tangent Kernels (NTK) and borrow tools from systems and control theory to analyze both the local and global training dynamics (e.g. stability, reachability). Second, we propose to dynamically alter the gradient descent training mechanism via fictitious labels as control inputs and an optimal state feedback policy. In this way, we enforce locally H2 optimal and convergent training behavior. The novel algorithm, Controlled Descent Training (CDT), guarantees local convergence. CDT unleashes new potentials in the analysis, interpretation, and design of ANN architectures. The applicability of the method is demonstrated on standard regression and classification problems.

onvergent learning

label augmentation

Neural Tangent Kernel

optimal labels

label selection

gradient decent training

Author

Viktor Andersson

Chalmers, Electrical Engineering, Systems and control

Centiro

Balázs Varga

Centiro

Vincent Szolnoky

Centiro

Chalmers, Mathematical Sciences, Applied Mathematics and Statistics

Andreas Syren

Centiro

Rebecka Jörnsten

Chalmers, Mathematical Sciences, Applied Mathematics and Statistics

Balázs Adam Kulcsár

Chalmers, Electrical Engineering, Systems and control

International Journal of Robust and Nonlinear Control

1049-8923 (ISSN) 1099-1239 (eISSN)

Vol. 34

Robustly and Optimally Controlled Training Of neural Networks II (OCTON II)

Centiro, 2020-05-01 -- 2025-04-30.

Robustly and Optimally Controlled Training Of neural Networks I (OCTON I)

Centiro, 2019-10-15 -- 2023-10-15.

Areas of Advance

Transport

Subject Categories

Mathematics

Computer and Information Science

Control Engineering

DOI

10.1002/rnc.7194

More information

Latest update

7/17/2024