Mutation Testing in Continuous Integration: An Exploratory Industrial Case Study
Paper i proceeding, 2023

Despite its potential quality benefits, the cost of mutation testing and the immaturity of mutation tools for many languages have led to a lack of adoption in industrial software development. In an exploratory case study at Zenseact - a company in the automotive domain - we have explored how mutation testing could be effectively applied in a typical Continuous Integration-based workflow. We evaluated the capabilities of C++ mutation tools, and demonstrate their use in GitHub Actionsbased CI workflows. Our investigation reveals that Dextool and Mull could be used in a CI workflow. Additionally, we conducted an interview study to understand how developers would use mutation testing in their CI workflows. Based on our qualitative analysis and practices proposed in literature, we discuss recommendations to integrate mutation testing in a CI workflow. For instance, visualising trends in the mutation score enable practitioners to understand how test quality is evolving. Moreover, tools should have a balance between offering fast feedback and keeping or flagging relevant mutants. Lastly, practitioners raised the need that the mutation should be applied at commit level, and that developers inexperienced with mutation testing should be trained in the implications of the practice.


Mutation Testing

Continuous Integration


Jonathan Orgard

Göteborgs universitet

Gregory Gay

Göteborgs universitet

Francisco Gomes

Göteborgs universitet

Kim Viggedal

Zenseact AB

Proceedings - 2023 IEEE 16th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2023

9798350333350 (ISBN)

16th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2023
Dublin, Ireland,




Datavetenskap (datalogi)





Mer information

Senast uppdaterat