Facing the truth: Benchmarking the techniques for the evolution of variant-rich systems
Paper in proceeding, 2019

The evolution of variant-rich systems is a challenging task. To support developers, the research community has proposed a range of different techniques over the last decades. However, many techniques have not been adopted in practice so far. To advance such techniques and to support their adoption, it is crucial to evaluate them against realistic baselines, ideally in the form of generally accessible benchmarks. To this end, we need to improve our empirical understanding of typical evolution scenarios for variant-rich systems and their relevance for benchmarking. In this paper, we establish eleven evolution scenarios in which benchmarks would be beneficial. Our scenarios cover typical lifecycles of variant-rich system, ranging from clone&own to adopting and evolving a configurable product-line platform. For each scenario, we formulate benchmarking requirements and assess its clarity and relevance via a survey with experts in variant-rich systems and software evolution. We also surveyed the existing benchmarking landscape, identifying synergies and gaps. We observed that most scenarios, despite being perceived as important by experts, are only partially or not at all supported by existing benchmarks-a call to arms for building community benchmarks upon our requirements. We hope that our work raises awareness for benchmarking as a means to advance techniques for evolving variant-rich systems, and that it will lead to a benchmarking initiative in our community.

Software variability


Product lines

Software evolution


Daniel Strüber

University of Gothenburg

Mukelabai Mukelabai

University of Gothenburg

Jacob Krüger

Otto von Guericke Universitaet Magdeburg

Stefan Fischer

Johannes Kepler University of Linz (JKU)

Lukas Linsbauer

Johannes Kepler University of Linz (JKU)

Jabier Martinez


Thorsten Berger

University of Gothenburg

ACM International Conference Proceeding Series

Vol. A 177-188
978-145037138-4 (ISBN)

23rd International Systems and Software Product Line Conference, SPLC 2019, co-located with the 13th European Conference on Software Architecture, ECSA 2019
Paris, France,

Subject Categories

Software Engineering

Information Science

Information Systemes, Social aspects



More information

Latest update