Evolution of statistical analysis in empirical software engineering research: Current state and steps forward
Artikel i vetenskaplig tidskrift, 2019
© 2019 Elsevier Inc. Software engineering research is evolving and papers are increasingly based on empirical data from a multitude of sources, using statistical tests to determine if and to what degree empirical evidence supports their hypotheses. To investigate the practices and trends of statistical analysis in empirical software engineering (ESE), this paper presents a review of a large pool of papers from top-ranked software engineering journals. First, we manually reviewed 161 papers and in the second phase of our method, we conducted a more extensive semi-automatic classification of papers spanning the years 2001–2015 and 5196 papers. Results from both review steps was used to: i) identify and analyse the predominant practices in ESE (e.g., using t-test or ANOVA), as well as relevant trends in usage of specific statistical methods (e.g., nonparametric tests and effect size measures) and, ii) develop a conceptual model for a statistical analysis workflow with suggestions on how to apply different statistical methods as well as guidelines to avoid pitfalls. Lastly, we confirm existing claims that current ESE practices lack a standard to report practical significance of results. We illustrate how practical significance can be discussed in terms of both the statistical analysis and in the practitioner's context.
Empirical software engineering
Practical significance
Statistical methods
Semi-automated literature review
Författare
Francisco Gomes
Chalmers, Data- och informationsteknik, Software Engineering, Software Engineering for Testing, Requirements, Innovation and Psychology
Göteborgs universitet
Richard Torkar
Chalmers, Data- och informationsteknik, Software Engineering, Software Engineering for Testing, Requirements, Innovation and Psychology
Göteborgs universitet
Robert Feldt
Göteborgs universitet
Chalmers, Data- och informationsteknik, Software Engineering, Software Engineering for Cyber Physical Systems
Blekinge Tekniska Högskola, BTH
Lucas Gren
Chalmers, Data- och informationsteknik
Göteborgs universitet
Carlo A Furia
Universita della Svizzera italiana
Zhongjie Huang
Göteborgs universitet
Chalmers, Mekanik och maritima vetenskaper, Strömningslära
Journal of Systems and Software
0164-1212 (ISSN)
Vol. 156 246-267Ämneskategorier
Annan data- och informationsvetenskap
Programvaruteknik
Systemvetenskap, informationssystem och informatik med samhällsvetenskaplig inriktning
DOI
10.1016/j.jss.2019.07.002