VICMus: Variance-Invariance-Covariance Regularization for Music Representation Learning

Paper i proceeding, 2024

Recent self-supervised learning methods often prevent informational collapse by implicitly regularizing information. Variance-Invariance-Covariance regularization (VICReg) was introduced as a non-contrastive loss function that explicitly maximizes information through regularization. While VICReg has garnered substantial interest in the field of computer vision, its application to the music domain remains unexplored. To address this gap, we introduce VICMus - VICReg for music representation learning. We pre-train VICMus on the Free Music Archive and achieve 36.3 mAP on MagnaTagaTune, outperforming Contrastive Learning of Musical Representations (CLMR), a recent contrastive method pre-trained on the ten times larger Million Song Dataset, which got 35.6 mAP. We evaluate VICMus on the Holistic Audio Representation Evaluation Suite (HARES)-music benchmark and achieve an average score of 51.7. Our results indicate that while VICMus may not yet achieve the performance of state-of-the-art self-supervised models, it offers a promising and computationally efficient avenue for music representation learning. Our code and models are available at https://github.com/SebastianLoef/VICMus.