Adaptation under Distributional Shifts in Centralized and Federated settings

Doctoral thesis, 2025

Using large datasets to train ever larger models in machine learning have
been extremely impactful for many diverse tasks. However, the challenge of
ensuring and predicting model generalization under distribution shifts remains
an open problem. Such shifts may occur between training and testing envir-
onments or even during the training process itself. In real-world applications,
these distribution changes can severely degrade model performance, making
adaptation a critical concern. This is the focus of domain adaptation (DA),
a field dedicated to developing both theoretical frameworks and methods for
settings with distribution shift. Domain adaptation primarily operates within
the supervised learning paradigm, where access to a large, centralized dataset
is assumed. However, such data availability is not always feasible due to
privacy concerns or the high costs associated with data collection and storage.
The federated learning (FL) setting addresses this by training models across
decentralized clients coordinated by a central server. Since clients retain local
data, distribution shifts, known as data heterogeneity, can arise between clients.
This may potentially degrade model performance. This thesis aims to overcome
some of these limitations in both the centralized and federated settings. In
particular, this is achieved by (i) questioning how to measure performance
under distribution shift in a practical way, (ii) proposing novel assumptions
and settings where we expand the amount of information available and (iii)
developing competitive methods for these settings. First, we explore the meas-
urement of performance in domain adaptation through evaluating theoretical
bounds. We survey the field of available domain adaptation bounds with an
eye towards their practicality and, after selecting candidates, make empirical
comparisons. Next, we consider a novel set of assumptions based on having
access to privileged information which we show is both practical and empirically
sound. We continue with expanding on the idea of additional information in
the FL setting where we show that access to label marginals can substantially
improve performance in cases where clients are meaningfully heterogeneous.
Finally, we explore another aspect of heterogeneity in FL where the label sets
of clients are non-identical and clients are unwilling to share them.