Controlled Descent Training

Licentiate thesis, 2023

In this work, a novel and model-based artificial neural network (ANN) train-
ing method is developed supported by optimal control theory. The method
augments training labels in order to robustly guarantee training loss conver-
gence and improve training convergence rate. Dynamic label augmentation
is proposed within the framework of gradient descent training where the con-
vergence of training loss is controlled. First, we capture the training behavior
with the help of empirical Neural Tangent Kernels (NTK) and borrow tools
from systems and control theory to analyze both the local and global training
dynamics (e.g. stability, reachability). Second, we propose to dynamically
alter the gradient descent training mechanism via fictitious labels as control
inputs and an optimal state feedback policy. In this way, we enforce locally H2
optimal and convergent training behavior. The novel algorithm, Controlled
Descent Training (CDT), guarantees local convergence. CDT unleashes new
potentials in the analysis, interpretation, and design of ANN architectures.
The applicability of the method is demonstrated on standard regression and
classification problems.