Performance of Regression Models as a Function of Experiment Noise
Artikel i vetenskaplig tidskrift, 2021

Background: A challenge in developing machine learning regression models is that it is difficult to know whether maximal performance has been reached on the test dataset, or whether further model improvement is possible. In biology, this problem is particularly pronounced as sample labels (response variables) are typically obtained through experiments and therefore have experiment noise associated with them. Such label noise puts a fundamental limit to the metrics of performance attainable by regression models on the test dataset. Results: We address this challenge by deriving an expected upper bound for the coefficient of determination (R2) for regression models when tested on the holdout dataset. This upper bound depends only on the noise associated with the response variable in a dataset as well as its variance. The upper bound estimate was validated via Monte Carlo simulations and then used as a tool to bootstrap performance of regression models trained on biological datasets, including protein sequence data, transcriptomic data, and genomic data. Conclusions: The new method for estimating upper bounds for model performance on test data should aid researchers in developing ML regression models that reach their maximum potential. Although we study biological datasets in this work, the new upper bound estimates will hold true for regression models from any research field or application area where response variables have associated noise.

machine learning

label noise

regression models

upper bound

experiment noise

Författare

Gang Li

Chalmers, Biologi och bioteknik, Systembiologi

Jan Zrimec

Chalmers, Biologi och bioteknik, Systembiologi

Boyang Ji

Danmarks Tekniske Universitet (DTU)

Chalmers, Biologi och bioteknik, Systembiologi

Jun Geng

Chalmers, Biologi och bioteknik, Systembiologi

Johan Larsbrink

Chalmers, Biologi och bioteknik, Industriell bioteknik

Aleksej Zelezniak

Science for Life Laboratory (SciLifeLab)

Chalmers, Biologi och bioteknik, Systembiologi

Jens B Nielsen

BioInnovation Institute

Chalmers, Biologi och bioteknik, Systembiologi

Danmarks Tekniske Universitet (DTU)

Martin Engqvist

Chalmers, Biologi och bioteknik, Systembiologi

Bioinformatics and Biology Insights

1177-9322 (ISSN)

Vol. 15

Predictive and Accelerated Metabolic Engineering Network (PAcMEN)

Europeiska kommissionen (EU) (EC/H2020/722287), 2016-09-01 -- 2020-08-30.

Ämneskategorier

Bioinformatik (beräkningsbiologi)

Sannolikhetsteori och statistik

Reglerteknik

DOI

10.1177/11779322211020315

Mer information

Senast uppdaterat

2021-07-05