Replications, Revisions, and Reanalyses: Managing Empirical Evidence in Software Engineering

Journal article, 2026

Background One aspired outcome of empirical research on quantitative data is a variance theory, i.e., a quantification of the effect of an independent on a dependent variables. In software engineering (SE) research, variance theories quantify-among others-the impact of tools, techniques, and other treatments on software development outcomes like productivity, cost-efficiency, and defect-detection rates. The validity of variance theories stems from the synthesis of multiple pieces of evidence, which increases its validity beyond the findings of a single study.
Gap However, research synthesis in SE is rare and-if done-mostly limited to purely narrative syntheses. At best, researchers perform meta-analyses to synthesize variance theories from several quantitative results. But even meta-analyses only produce reliable results when synthesizing exact replications yet fail to generalize from variations.
Goal We aim to extend the frontier of research synthesis beyond the state-of-the-art to systematically manage empirical evidence and its evolution.
Method We apply method engineering to construct a framework for research synthesis from proven, individual method fragments. The framework allows researchers to put new evidence in a clear relation to an existing body of evidence and systematically expand knowledge about a studied phenomenon. We demonstrate the application of this framework to two fields of research by explicitly modeling the relationship between existing pieces of evidence.
Result The framework puts three types of evolution of evidence into relation: (1) replications investigate the same hypothesis in a new context to improve external validity, (2) revisions challenge an existing hypothesis to improve internal validity, and (3) reanalyses replace analysis methods to improve conclusion validity. Through a systematic evolution of evidence and clear assessment criteria for each dimension of validity, the proposed framework can determine the frontier of a field of research.
Conclusion The framework provides a perspective to systematically evolve empirical evidence in SE, supporting more constructive and productive advances in our field.