Evolution of statistical analysis in empirical software engineering research: Current state and steps forward
Artikel i vetenskaplig tidskrift, 2019

© 2019 Elsevier Inc. Software engineering research is evolving and papers are increasingly based on empirical data from a multitude of sources, using statistical tests to determine if and to what degree empirical evidence supports their hypotheses. To investigate the practices and trends of statistical analysis in empirical software engineering (ESE), this paper presents a review of a large pool of papers from top-ranked software engineering journals. First, we manually reviewed 161 papers and in the second phase of our method, we conducted a more extensive semi-automatic classification of papers spanning the years 2001–2015 and 5196 papers. Results from both review steps was used to: i) identify and analyse the predominant practices in ESE (e.g., using t-test or ANOVA), as well as relevant trends in usage of specific statistical methods (e.g., nonparametric tests and effect size measures) and, ii) develop a conceptual model for a statistical analysis workflow with suggestions on how to apply different statistical methods as well as guidelines to avoid pitfalls. Lastly, we confirm existing claims that current ESE practices lack a standard to report practical significance of results. We illustrate how practical significance can be discussed in terms of both the statistical analysis and in the practitioner's context.

Empirical software engineering

Practical significance

Statistical methods

Semi-automated literature review

Författare

Francisco Gomes

Chalmers, Data- och informationsteknik, Software Engineering, Software Engineering for Testing, Requirements, Innovation and Psychology

Göteborgs universitet

Forskning Andra publikationer

Richard Torkar

Chalmers, Data- och informationsteknik, Software Engineering, Software Engineering for Testing, Requirements, Innovation and Psychology

Göteborgs universitet

Forskning Andra publikationer

Robert Feldt

Göteborgs universitet

Chalmers, Data- och informationsteknik, Software Engineering, Software Engineering for Cyber Psysical Systems

Blekinge Tekniska Högskola, BTH

Forskning Andra publikationer

Lucas Gren

Chalmers, Data- och informationsteknik

Göteborgs universitet

Forskning Andra publikationer

Carlo A Furia

Universita della Svizzera italiana

Forskning Andra publikationer

Zhongjie Huang

Göteborgs universitet

Chalmers, Mekanik och maritima vetenskaper, Strömningslära

Forskning Andra publikationer

Journal of Systems and Software

0164-1212 (ISSN)

Vol. 156 246-267

Ämneskategorier

Annan data- och informationsvetenskap

Programvaruteknik

Systemvetenskap, informationssystem och informatik med samhällsvetenskaplig inriktning

DOI

10.1016/j.jss.2019.07.002

Publikationsdata kopplat till DOI

Mer information

Senast uppdaterat

2019-08-22

Feedback och support

Om du har frågor, behöver hjälp, hittar en bugg eller vill ge feedback kan du göra det här nedan. Du når oss också direkt per e-post research.lib@chalmers.se eller via telefon 031-772 3744.

Om tjänsten

Research.chalmers.se innehåller information om forskning på Chalmers, publikationer och projekt inklusive information om finansiärer och samarbetspartners.

Läs mer om tjänsten, täckningsgrad och vilka som kan se informationen

Personuppgifter och cookies

Länkar

Chalmers bibliotek
Chalmers forskning
Forskningsprojekt på chalmers.se

Chalmers tekniska högskola

412 96 GÖTEBORG
TELEFON: 031-772 10 00
WWW.CHALMERS.SE