Statistical and practical significance of empirical software engineering research: A maturity model

Preprint, 2017

Software engineering research is maturing and papers increasingly support their arguments with empirical data from a multitude of sources, using statistical tests to judge if and to what degree empirical evidence supports their hypotheses. This paper presents trends and current state of art regarding statistical maturity in empirical software engineering research; the objective being the development of a statistical maturity model. First, we manually reviewed papers from four well-known and top ranked journals producing a review protocol along with the view of current (2015) state of art concerning statistical maturity, practical significance and reproducibility of empirical software engineering research. Our protocol was then used as ground truth, i.e., training set, for a semi-automatic classification of studies for the years 2001--2015 using a total of 3,011 papers. We used the extracted data to develop a statistical maturity model which also includes a model for how to argue for practical significance. The statistical maturity of empirical software engineering research has an upward trend in certain areas (e.g., use of nonparametric statistics, but also more generally in the usage of quantitative analysis). However, we also see how our research area currently often fails to connect the statistical analysis to practical significance. For instance, we argue that conclusions should explicitly state contributions to software engineering practice, e.g., the return on investment for practitioners. We argue that the statistical maturity model can be used by researchers and practitioners to build a coherent statistical analysis and guide them in the choice of statistical approaches of its steps. The final goal for a researcher would be to, in a clearer way, present and argue for the practical significance of their findings. Bayesian analysis, we believe, has a role to play in this.