Do Kernel and Neural Embeddings Help in Training and Generalization?
Artikel i vetenskaplig tidskrift, 2023

Recent results on optimization and generalization properties of neural networks showed that in a simple two-layer network, the alignment of the labels to the eigenvectors of the corresponding Gram matrix determines the convergence of the optimization during training. Such analyses also provide upper bounds on the generalization error. We experimentally investigate the implications of these results to deeper networks via embeddings. We regard the layers preceding the final hidden layer as producing different representations of the input data which are then fed to the two-layer model. We show that these representations improve both optimization and generalization. In particular, we investigate three kernel representations when fed to the final hidden layer: the Gaussian kernel and its approximation by random Fourier features, kernels designed to imitate representations produced by neural networks and finally an optimal kernel designed to align the data with target labels. The approximated representations induced by these kernels are fed to the neural network and the optimization and generalization properties of the final model are evaluated and compared.

Gram matrix

Convergence

Kernel embedding

Neural Network

Författare

Arman Rahbar

Chalmers, Data- och informationsteknik, Data Science och AI

Emilio Jorge

Chalmers, Data- och informationsteknik, Data Science och AI

Devdatt Dubhashi

Chalmers, Data- och informationsteknik, Data Science och AI

Morteza Haghir Chehreghani

Chalmers, Data- och informationsteknik, Data Science och AI

Neural Processing Letters

1370-4621 (ISSN) 1573-773X (eISSN)

Vol. 55 2 1681-1695

Ämneskategorier

Språkteknologi (språkvetenskaplig databehandling)

Telekommunikation

Bioinformatik (beräkningsbiologi)

Sannolikhetsteori och statistik

DOI

10.1007/s11063-022-10958-8

Mer information

Senast uppdaterat

2023-07-05