Evaluated CMI Bounds for Meta Learning: Tightness and Expressiveness
Paper i proceeding, 2022

Recent work has established that the conditional mutual information (CMI) framework of Steinke and Zakynthinou (2020) is expressive enough to capture generalization guarantees in terms of algorithmic stability, VC dimension, and related complexity measures for conventional learning (Harutyunyan et al., 2021, Haghifam et al., 2021). Hence, it provides a unified method for establishing generalization bounds. In meta learning, there has so far been a divide between information-theoretic results and results from classical learning theory. In this work, we take a first step toward bridging this divide. Specifically, we present novel generalization bounds for meta learning in terms of the evaluated CMI (e-CMI). To demonstrate the expressiveness of the e-CMI framework, we apply our bounds to a representation learning setting, with $n$ samples from $\hat n$ tasks parameterized by functions of the form $f_i \circ h$. Here, each $f_i \in \mathcal F$ is a task-specific function, and $h \in \mathcal H$ is the shared representation. For this setup, we show that the e-CMI framework yields a bound that scales as $\sqrt{ \mathcal C(\mathcal H)/(n\hat n) + \mathcal C(\mathcal F)/n} $, where $\mathcal C(\cdot)$ denotes a complexity measure of the hypothesis class. This scaling behavior coincides with the one reported in Tripuraneni et al. (2020) using Gaussian complexity.

information theory

generalization bounds

meta learning

machine learning


Fredrik Hellström

Chalmers, Elektroteknik, Kommunikation, Antenner och Optiska Nätverk

Giuseppe Durisi

Chalmers, Elektroteknik, Kommunikation, Antenner och Optiska Nätverk

Advances in Neural Information Processing Systems

10495258 (ISSN)

Vol. 35
9781713871088 (ISBN)

Conference on Neural Information Processing Systems (NeurIPS)
New Orleans, USA,

INNER: information theory of deep neural networks

Chalmers AI-forskningscentrum (CHAIR), 2019-01-01 -- 2021-12-31.


Informations- och kommunikationsteknik


Annan matematik

Datorseende och robotik (autonoma system)

Matematisk analys



Mer information

Senast uppdaterat