The Comparative Evaluation of Test Prioritization Approaches in an Industrial Study

Paper i proceeding, 2023

Many test prioritisation techniques have been proposed in order to improve test effectiveness of Continuous Integration (CI) pipelines. Particularly, diversity-based testing (DBT) has shown promising and competitive results to improve test effectiveness. We report on a case study considering the CI pipeline of Axis Communications in Sweden. We compared three different prioritisation approaches (i.e., diversity, failure history and time) in terms of their impact on coverage, failure detection rates and reduction on test execution time. Our results reveal that DBT is the best candidate to provide feature coverage, whereas failure rate prioritisation yields better failure coverage. Time-based prioritisation is not a reliable approach to provide cost-effective testing. Moreover, DBT would allow stakeholders to receive quick feedback on many combinations of integrated features to verify their code changes. Our participants report that developers are mainly interested in: (i) receiving quick feedback on a high combination of integrated features to verify their code changes, and (ii) associate their test suites to confidence scores representing the risk of missing failures given that fewer tests are executed.