On the Interpretability of Regularisation for Neural Networks Through Model Gradient Similarity
Paper in proceeding, 2022

Most complex machine learning and modelling techniques are prone to overfitting and may subsequently generalise poorly to future data. Artificial neural networks are no different in this regard and, despite having a level of implicit regularisation when trained with gradient descent, often require the aid of explicit regularisers. We introduce a new framework, Model Gradient Similarity (MGS), that (1) serves as a metric of regularisation, which can be used to monitor neural network training, (2) adds insight into how explicit regularisers, while derived from widely different principles, operate via the same mechanism underneath by increasing MGS, and (3) provides the basis for a new regularisation scheme which exhibits excellent performance, especially in challenging settings such as high levels of label noise or limited sample sizes.

Author

Vincent Szolnoky

Chalmers, Mathematical Sciences, Applied Mathematics and Statistics

Viktor Andersson

Chalmers, Electrical Engineering, Systems and control

Balázs Adam Kulcsár

Chalmers, Electrical Engineering, Systems and control

Rebecka Jörnsten

Chalmers, Mathematical Sciences, Applied Mathematics and Statistics

Advances in Neural Information Processing Systems

10495258 (ISSN)

Vol. 35
9781713871088 (ISBN)

36th Conference on Neural Information Processing Systems, NeurIPS 2022
New Orleans, USA,

Subject Categories

Communication Systems

Bioinformatics (Computational Biology)

Computer Systems

ISBN

9781713871088

More information

Latest update

1/16/2024