Conditional Mutual Information-Based Generalization Bound for Meta Learning
Paper i proceeding, 2021

Meta-learning optimizes an inductive bias—typically in the form of the hyperparameters of a base-learning algorithm—by observing data from a finite number of related tasks. This paper presents an information-theoretic bound on the generalization performance of any given meta-learner, which builds on the conditional mutual information (CMI) framework of Steinke and Zakynthinou (2020). In the proposed extension to meta-learning, the CMI bound involves a training meta-supersample obtained by first sampling 2N independent tasks from the task environment, and then drawing 2M independent training samples for each sampled task. The meta-training data fed to the meta-learner is modelled as being obtained by randomly selecting N tasks from the available 2N tasks and M training samples per task from the available 2M training samples per task. The resulting bound is explicit in two CMI terms, which measure the information that the meta-learner output and the base-learner output provide about which training data are selected, given the entire meta-supersample. Finally, we present a numerical example that illustrates the merits of the proposed bound in comparison to prior information-theoretic bounds for meta-learning

Författare

Arezou Rezazadeh

Chalmers, Elektroteknik, Kommunikations- och antennsystem, Kommunikationssystem

S.T. Jose

King's College London

Giuseppe Durisi

Chalmers, Elektroteknik, Kommunikations- och antennsystem, Kommunikationssystem

Osvaldo Simeone

IEEE International Symposium on Information Theory - Proceedings

21578095 (ISSN)

1176-1181

2021 IEEE International Symposium on Information Theory, ISIT 2021
Melbourne, Australia,

Towards a greater understanding of deep neural networks (IT-DNN)

Europeiska kommissionen (EU) (EC/H2020/893082), 2020-04-01 -- 2022-03-31.

Ämneskategorier

Annan data- och informationsvetenskap

Lärande

Sannolikhetsteori och statistik

DOI

10.1109/ISIT45174.2021.9518020

Mer information

Senast uppdaterat

2021-09-28