Applying test case prioritization to software microbenchmarks
Journal article, 2021

Regression testing comprises techniques which are applied during software evolution to uncover faults effectively and efficiently. While regression testing is widely studied for functional tests, performance regression testing, e.g., with software microbenchmarks, is hardly investigated. Applying test case prioritization (TCP), a regression testing technique, to software microbenchmarks may help capturing large performance regressions sooner upon new versions. This may especially be beneficial for microbenchmark suites, because they take considerably longer to execute than unit test suites. However, it is unclear whether traditional unit testing TCP techniques work equally well for software microbenchmarks. In this paper, we empirically study coverage-based TCP techniques, employing total and additional greedy strategies, applied to software microbenchmarks along multiple parameterization dimensions, leading to 54 unique technique instantiations. We find that TCP techniques have a mean APFD-P (average percentage of fault-detection on performance) effectiveness between 0.54 and 0.71 and are able to capture the three largest performance changes after executing 29% to 66% of the whole microbenchmark suite. Our efficiency analysis reveals that the runtime overhead of TCP varies considerably depending on the exact parameterization. The most effective technique has an overhead of 11% of the total microbenchmark suite execution time, making TCP a viable option for performance regression testing. The results demonstrate that the total strategy is superior to the additional strategy. Finally, dynamic-coverage techniques should be favored over static-coverage techniques due to their acceptable analysis overhead; however, in settings where the time for prioritzation is limited, static-coverage techniques provide an attractive alternative.

JMH

performance testing

software microbenchmarking

regression testing

test case prioritization

Author

Christoph Laaber

University of Zürich

Harald C. Gall

University of Zürich

Philipp Leitner

Cyber Physical Systems

Empirical Software Engineering

1382-3256 (ISSN) 1573-7616 (eISSN)

Vol. 26 6 133

ImmeRSEd - Developer-Targeted Performance Engineering for Immersed Release and Software Engineers

Swedish Research Council (VR) (2018-04127), 2019-01-01 -- 2023-12-31.

Subject Categories

Computer Engineering

Software Engineering

Computer Science

DOI

10.1007/s10664-021-10037-x

More information

Latest update

11/3/2023