Effortless Training of Joint Energy-Based Models with Sliced Score Matching
Paper in proceeding, 2022

Standard discriminative classifiers can be upgraded to joint energy-based models (JEMs) by combining the classification loss with a log-evidence loss. Hence, such models intrinsically allow detection of out-of-distribution (OOD) samples, and empirically also provide better-calibrated posteriors, i.e., prediction uncertainties.
However, the training procedure suggested for JEMs (using stochastic gradient Langevin dynamics---or SGLD---to maximize the evidence) is reported to be brittle.
In this work, we propose to utilize score matching---in particular sliced score matching---to obtain a stable training method for JEMs. We observe empirically that the combination of score matching with the standard classification loss leads to improved OOD detection and better-calibrated classifiers for otherwise identical DNN architectures.
Additionally, we also analyze the impact of replacing the regular soft-max layer for classification with a gated soft-max one in order to improve the intrinsic transformation invariance and generalization ability.

Author

Xixi Liu

Chalmers, Electrical Engineering, Signal Processing and Biomedical Engineering

Dorian Staudt

Chalmers, Electrical Engineering, Signal Processing and Biomedical Engineering

Che-Tsung Lin

Chalmers, Electrical Engineering, Signal Processing and Biomedical Engineering

Christopher Zach

Chalmers, Electrical Engineering, Signal Processing and Biomedical Engineering

Proceedings - International Conference on Pattern Recognition

10514651 (ISSN)

2643-2649
978-1-6654-9062-7 (ISBN)

26th International Conference on Pattern Recognition, ICPR 2022
Montréal Québec, Canada,

Subject Categories

Probability Theory and Statistics

Computer Vision and Robotics (Autonomous Systems)

DOI

10.1109/ICPR56361.2022.9956495

ISBN

9781665490627

More information

Latest update

1/3/2024 9