Controlled Descent Training
Artikel i vetenskaplig tidskrift, 2024

In this work, a novel and model-based artificial neural network (ANN) training method is developed supported by optimal control theory. The method augments training labels in order to robustly guarantee training loss convergence and improve training convergence rate. Dynamic label augmentation is proposed within the framework of gradient descent training where the convergence of training loss is controlled. First, we capture the training behavior with the help of empirical Neural Tangent Kernels (NTK) and borrow tools from systems and control theory to analyze both the local and global training dynamics (e.g. stability, reachability). Second, we propose to dynamically alter the gradient descent training mechanism via fictitious labels as control inputs and an optimal state feedback policy. In this way, we enforce locally H2 optimal and convergent training behavior. The novel algorithm, Controlled Descent Training (CDT), guarantees local convergence. CDT unleashes new potentials in the analysis, interpretation, and design of ANN architectures. The applicability of the method is demonstrated on standard regression and classification problems.

onvergent learning

label augmentation

Neural Tangent Kernel

optimal labels

label selection

gradient decent training

Författare

Viktor Andersson

Chalmers, Elektroteknik, System- och reglerteknik

Centiro

Balázs Varga

Centiro

Vincent Szolnoky

Centiro

Chalmers, Matematiska vetenskaper, Tillämpad matematik och statistik

Andreas Syren

Centiro

Rebecka Jörnsten

Chalmers, Matematiska vetenskaper, Tillämpad matematik och statistik

Balázs Adam Kulcsár

Chalmers, Elektroteknik, System- och reglerteknik

International Journal of Robust and Nonlinear Control

1049-8923 (ISSN) 1099-1239 (eISSN)

Vol. 34

Robustly and Optimally Controlled Training Of neural Networks II (OCTON II)

Centiro, 2020-05-01 -- 2025-04-30.

Robustly and Optimally Controlled Training Of neural Networks I (OCTON I)

Centiro, 2019-10-15 -- 2023-10-15.

Styrkeområden

Transport

Ämneskategorier

Matematik

Data- och informationsvetenskap

Reglerteknik

DOI

10.1002/rnc.7194

Mer information

Senast uppdaterat

2024-07-17