Transfer Meta-Learning: Information-Theoretic Bounds and Information Meta-Risk Minimization
Artikel i vetenskaplig tidskrift, 2021

Meta-learning automatically infers an inductive bias by observing data from a number of related tasks. The inductive bias is encoded by hyperparameters that determine aspects of the model class or training algorithm, such as initialization or learning rate. Meta-learning assumes that the learning tasks belong to a task environment, and that tasks are drawn from the same task environment both during meta-training and meta-testing. This, however, may not hold true in practice. In this paper, we introduce the problem of transfer meta-learning, in which tasks are drawn from a target task environment during meta-testing that may differ from the source task environment observed during meta-training. Novel information-theoretic upper bounds are obtained on the transfer meta-generalization gap, which measures the difference between the meta-training loss, available at the meta-learner, and the average loss on meta-test data from a new, randomly selected, task in the target task environment. The first bound, on the average transfer meta-generalization gap, captures the meta-environment shift between source and target task environments via the KL divergence between source and target data distributions. The second, PAC-Bayesian bound, and the third, single-draw bound, account for this shift via the log-likelihood ratio between source and target task distributions. Furthermore, two transfer meta-learning solutions are introduced. For the first, termed Empirical Meta-Risk Minimization (EMRM), we derive bounds on the average optimality gap. The second, referred to as Information Meta-Risk Minimization (IMRM), is obtained by minimizing the PAC-Bayesian bound. IMRM is shown via experiments to potentially outperform EMRM.

Loss measurement

Training

Risk management

information-theoretic generalization bounds

PAC-Bayesian bounds

Task analysis

Hospitals

Transfer learning

single-draw bounds

Transfer meta-learning

Upper bound

information risk minimization

Författare

Sharu Theresa Jose

King's College London

Osvaldo Simeone

King's College London

Giuseppe Durisi

Chalmers, Elektroteknik, Kommunikations- och antennsystem, Kommunikationssystem

IEEE Transactions on Information Theory

0018-9448 (ISSN)

Vol. In Press

Ämneskategorier

Annan data- och informationsvetenskap

Psykologi (exklusive tillämpad psykologi)

Lärande

DOI

10.1109/TIT.2021.3119605

Mer information

Senast uppdaterat

2021-11-02