What's Wrong With My Benchmark Results? Studying Bad Practices in JMH Benchmarks
Journal article, 2021

Microbenchmarking frameworks, such as Java's Microbenchmark Harness (JMH), allow developers to write fine-grained performance test suites at the method or statement level. However, due to the complexities of the Java Virtual Machine, developers often struggle with writing expressive JMH benchmarks which accurately represent the performance of such methods or statements. In this paper, we empirically study bad practices of JMH benchmarks. We present a tool that leverages static analysis to identify 5 bad JMH practices. Our empirical study of 123 open source Java-based systems shows that each of these 5 bad practices are prevalent in open source software. Further, we conduct several experiments to quantify the impact of each bad practice in multiple case studies, and find that bad practices often significantly impact the benchmark results. To validate our experimental results, we constructed patches that fix the identified bad practices for six of the studied open source projects, of which five were merged into the main branch of the project. In this paper, we show that developers struggle with accurate Java microbenchmarking, and provide several recommendations to developers of microbenchmarking frameworks on how to improve future versions of their framework.

Performance testing

Benchmark testing

Java

Static analysis

microbenchmarking

Optimization

static analysis

bad practices

JMH

Author

Diego Elias Damasceno Costa

Heidelberg University

Cor Paul Bezemer

Queen's University

Philipp Leitner

Chalmers, Computer Science and Engineering (Chalmers), Software Engineering (Chalmers)

Artur Andrzejak

Heidelberg University

IEEE Transactions on Software Engineering

0098-5589 (ISSN) 19393520 (eISSN)

Vol. 47 7 1452-1467 8747433

ImmeRSEd - Developer-Targeted Performance Engineering for Immersed Release and Software Engineers

Swedish Research Council (VR) (2018-04127), 2019-01-01 -- 2023-12-31.

Areas of Advance

Information and Communication Technology

Subject Categories

Software Engineering

DOI

10.1109/TSE.2019.2925345

More information

Latest update

8/17/2021