Conditional Mutual Information-Based Generalization Bound for Meta Learning
Paper in proceeding, 2021

Meta-learning optimizes an inductive bias—typically in the form of the hyperparameters of a base-learning algorithm—by observing data from a finite number of related tasks. This paper presents an information-theoretic bound on the generalization performance of any given meta-learner, which builds on the conditional mutual information (CMI) framework of Steinke and Zakynthinou (2020). In the proposed extension to meta-learning, the CMI bound involves a training meta-supersample obtained by first sampling 2N independent tasks from the task environment, and then drawing 2M independent training samples for each sampled task. The meta-training data fed to the meta-learner is modelled as being obtained by randomly selecting N tasks from the available 2N tasks and M training samples per task from the available 2M training samples per task. The resulting bound is explicit in two CMI terms, which measure the information that the meta-learner output and the base-learner output provide about which training data are selected, given the entire meta-supersample. Finally, we present a numerical example that illustrates the merits of the proposed bound in comparison to prior information-theoretic bounds for meta-learning

Author

Arezou Rezazadeh

Chalmers, Electrical Engineering, Communication and Antenna Systems, Communication Systems

S.T. Jose

King's College London

Giuseppe Durisi

Chalmers, Electrical Engineering, Communication and Antenna Systems, Communication Systems

Osvaldo Simeone

IEEE International Symposium on Information Theory - Proceedings

21578095 (ISSN)

1176-1181
9781538682098 (ISBN)

2021 IEEE International Symposium on Information Theory, ISIT 2021
Melbourne, Australia,

Towards a greater understanding of deep neural networks (IT-DNN)

European Commission (EC) (EC/H2020/893082), 2020-04-01 -- 2022-03-31.

Subject Categories

Other Computer and Information Science

Learning

Probability Theory and Statistics

DOI

10.1109/ISIT45174.2021.9518020

More information

Latest update

9/28/2021