Statistical Models for the Analysis of Optimization Algorithms with Benchmark Functions
Journal article, 2021

Frequentist statistical methods, such as hypothesis testing, are standard practices in studies that provide benchmark comparisons. Unfortunately, these methods have often been misused, e.g., without testing for their statistical test assumptions or without controlling for family-wise errors in multiple group comparisons, among several other problems. Bayesian Data Analysis (BDA) addresses many of the previously mentioned shortcomings but its use is not widely spread in the analysis of empirical data in the evolutionary computing community. This paper provides three main contributions. First, we motivate the need for utilizing Bayesian data analysis and provide an overview of this topic. Second, we discuss the practical aspects of BDA to ensure that our models are valid and the results are transparent. Finally, we provide five statistical models that can be used to answer multiple research questions. The online appendix provides a step-by-step guide on how to perform the analysis of the models discussed in this paper, including the code for the statistical models, the data transformations, and the discussed tables and figures.

Benchmark Comparison

Black-Box Optimization

Bayesian Data Analysis

Statistical models

Author

David Issa Mattos

Chalmers, Computer Science and Engineering (Chalmers), Software Engineering (Chalmers), Software Engineering for Cyber Physical Systems

Jan Bosch

Chalmers, Computer Science and Engineering (Chalmers), Software Engineering (Chalmers), Software Engineering for Testing, Requirements, Innovation and Psychology

Helena Holmström Olsson

Malmö university

IEEE Transactions on Evolutionary Computation

1089-778X (ISSN)

Vol. In Press

Subject Categories

Other Computer and Information Science

Bioinformatics (Computational Biology)

Probability Theory and Statistics

DOI

10.1109/TEVC.2021.3081167

More information

Latest update

6/23/2021