Outliers and Replication in Software Engineering
Paper in proceeding, 2014

Empirical software engineering is a research field of growing interest. Studies within this field handles an increasing amount of data. In order to replicate a study the data needs to be accessible and all processing of this data needs to be reproducible. Specifically, the handling of deviating data points, also known as outliers, needs to be documented in order for a study to be replicated. This study investigated the data availability for recently published studies within empirical software engineering. Furthermore, it also investigated if outliers are documented in the same research field. Papers were reviewed using a literature review and the presence of outliers was investigated using an unsupervised outlier detection method. Only 37% of the papers reviewed had their data accessible. Furthermore, in many cases outliers were present in the reviewed studies but 63% of the papers studies did not mention how outliers were handled. The data availability within empirical software engineering research is low and is hindering replication of studies. Additionally, the lack of documentation regarding how outliers are handled is hindering replication.

Empirical Software Engineering

Data handling

Outlier Detection


Data availability


Software engineering



Literature reviews


Erik Lindqvist

Student at Chalmers

Henrik Larsson

Student at Chalmers

Richard Torkar

University of Gothenburg

Proceedings - Asia-Pacific Software Engineering Conference, APSEC

15301362 (ISSN)


Subject Categories

Software Engineering



More information

Latest update

1/3/2024 9