Conditional Mutual Information-Based Generalization Bound for Meta Learning
Paper i proceeding, 2021

Meta-learning optimizes an inductive bias—typically in the form of the hyperparameters of a base-learning algorithm—by observing data from a finite number of related tasks. This paper presents an information-theoretic bound on the generalization performance of any given meta-learner, which builds on the conditional mutual information (CMI) framework of Steinke and Zakynthinou (2020). In the proposed extension to meta-learning, the CMI bound involves a training meta-supersample obtained by first sampling 2N independent tasks from the task environment, and then drawing 2M independent training samples for each sampled task. The meta-training data fed to the meta-learner is modelled as being obtained by randomly selecting N tasks from the available 2N tasks and M training samples per task from the available 2M training samples per task. The resulting bound is explicit in two CMI terms, which measure the information that the meta-learner output and the base-learner output provide about which training data are selected, given the entire meta-supersample. Finally, we present a numerical example that illustrates the merits of the proposed bound in comparison to prior information-theoretic bounds for meta-learning

Författare

Arezou Rezazadeh

Chalmers, Elektroteknik, Kommunikation, Antenner och Optiska Nätverk

S.T. Jose

King's College London

Giuseppe Durisi

Chalmers, Elektroteknik, Kommunikation, Antenner och Optiska Nätverk

Osvaldo Simeone

IEEE International Symposium on Information Theory - Proceedings

21578095 (ISSN)

Vol. 2021-July 1176-1181
9781538682098 (ISBN)

2021 IEEE International Symposium on Information Theory, ISIT 2021
Melbourne, Australia,

Towards a greater understanding of deep neural networks (IT-DNN)

Europeiska kommissionen (EU) (EC/H2020/893082), 2020-04-01 -- 2022-03-31.

Ämneskategorier

Annan data- och informationsvetenskap

Lärande

Sannolikhetsteori och statistik

DOI

10.1109/ISIT45174.2021.9518020

Mer information

Senast uppdaterat

2022-04-05