Facing the truth: Benchmarking the techniques for the evolution of variant-rich systems
Paper i proceeding, 2019

The evolution of variant-rich systems is a challenging task. To support developers, the research community has proposed a range of different techniques over the last decades. However, many techniques have not been adopted in practice so far. To advance such techniques and to support their adoption, it is crucial to evaluate them against realistic baselines, ideally in the form of generally accessible benchmarks. To this end, we need to improve our empirical understanding of typical evolution scenarios for variant-rich systems and their relevance for benchmarking. In this paper, we establish eleven evolution scenarios in which benchmarks would be beneficial. Our scenarios cover typical lifecycles of variant-rich system, ranging from clone&own to adopting and evolving a configurable product-line platform. For each scenario, we formulate benchmarking requirements and assess its clarity and relevance via a survey with experts in variant-rich systems and software evolution. We also surveyed the existing benchmarking landscape, identifying synergies and gaps. We observed that most scenarios, despite being perceived as important by experts, are only partially or not at all supported by existing benchmarks-a call to arms for building community benchmarks upon our requirements. We hope that our work raises awareness for benchmarking as a means to advance techniques for evolving variant-rich systems, and that it will lead to a benchmarking initiative in our community.

Benchmark

Product lines

Software evolution

Software variability

Författare

Daniel Strüber

Chalmers, Data- och informationsteknik, Software Engineering, Software Engineering for Cyber Physical Systems

Mukelabai Mukelabai

Chalmers, Data- och informationsteknik, Software Engineering, Software Engineering for Cyber Physical Systems

Jacob Krüger

Otto von Guericke Universitaet Magdeburg

Stefan Fischer

Johannes Kepler Universität Linz (JKU)

Lukas Linsbauer

Johannes Kepler Universität Linz (JKU)

Jabier Martinez

Tecnalia

Thorsten Berger

Chalmers, Data- och informationsteknik, Software Engineering, Software Engineering for Cyber Physical Systems

ACM International Conference Proceeding Series

Vol. A 177-188

23rd International Systems and Software Product Line Conference, SPLC 2019, co-located with the 13th European Conference on Software Architecture, ECSA 2019
Paris, France,

Ämneskategorier

Programvaruteknik

Systemvetenskap

Systemvetenskap, informationssystem och informatik med samhällsvetenskaplig inriktning

DOI

10.1145/3336294.3336302

Mer information

Senast uppdaterat

2020-06-14