Effortless Training of Joint Energy-Based Models with Sliced Score Matching
Paper i proceeding, 2022

Standard discriminative classifiers can be upgraded to joint energy-based models (JEMs) by combining the classification loss with a log-evidence loss. Hence, such models intrinsically allow detection of out-of-distribution (OOD) samples, and empirically also provide better-calibrated posteriors, i.e., prediction uncertainties.
However, the training procedure suggested for JEMs (using stochastic gradient Langevin dynamics---or SGLD---to maximize the evidence) is reported to be brittle.
In this work, we propose to utilize score matching---in particular sliced score matching---to obtain a stable training method for JEMs. We observe empirically that the combination of score matching with the standard classification loss leads to improved OOD detection and better-calibrated classifiers for otherwise identical DNN architectures.
Additionally, we also analyze the impact of replacing the regular soft-max layer for classification with a gated soft-max one in order to improve the intrinsic transformation invariance and generalization ability.

Författare

Xixi Liu

Chalmers, Elektroteknik, Signalbehandling och medicinsk teknik

Dorian Staudt

Chalmers, Elektroteknik, Signalbehandling och medicinsk teknik

Che-Tsung Lin

Chalmers, Elektroteknik, Signalbehandling och medicinsk teknik

Christopher Zach

Chalmers, Elektroteknik, Signalbehandling och medicinsk teknik

Proceedings - International Conference on Pattern Recognition

10514651 (ISSN)

Vol. 2022-August 2643-2649
978-1-6654-9062-7 (ISBN)

26th International Conference on Pattern Recognition, ICPR 2022
Montréal Québec, Canada,

Ämneskategorier

Sannolikhetsteori och statistik

Datorseende och robotik (autonoma system)

DOI

10.1109/ICPR56361.2022.9956495

ISBN

9781665490627

Mer information

Senast uppdaterat

2024-07-17