On the Interpretability of Regularisation for Neural Networks Through Model Gradient Similarity
Paper i proceeding, 2022

Most complex machine learning and modelling techniques are prone to overfitting and may subsequently generalise poorly to future data. Artificial neural networks are no different in this regard and, despite having a level of implicit regularisation when trained with gradient descent, often require the aid of explicit regularisers. We introduce a new framework, Model Gradient Similarity (MGS), that (1) serves as a metric of regularisation, which can be used to monitor neural network training, (2) adds insight into how explicit regularisers, while derived from widely different principles, operate via the same mechanism underneath by increasing MGS, and (3) provides the basis for a new regularisation scheme which exhibits excellent performance, especially in challenging settings such as high levels of label noise or limited sample sizes.

Författare

Vincent Szolnoky

Chalmers, Matematiska vetenskaper, Tillämpad matematik och statistik

Viktor Andersson

Chalmers, Elektroteknik, System- och reglerteknik

Balázs Adam Kulcsár

Chalmers, Elektroteknik, System- och reglerteknik

Rebecka Jörnsten

Chalmers, Matematiska vetenskaper, Tillämpad matematik och statistik

Advances in Neural Information Processing Systems

10495258 (ISSN)

Vol. 35
9781713871088 (ISBN)

36th Conference on Neural Information Processing Systems, NeurIPS 2022
New Orleans, USA,

Ämneskategorier

Kommunikationssystem

Bioinformatik (beräkningsbiologi)

Datorsystem

ISBN

9781713871088

Mer information

Senast uppdaterat

2024-01-16