Do Judge a Test by its Cover: Combining Combinatorial and Property-Based Testing
Paper i proceeding, 2021

Property-based testing uses randomly generated inputs to validate high-level program specifications. It can be shockingly effective at finding bugs, but it often requires generating a very large number of inputs to do so. In this paper, we apply ideas from combinatorial testing, a powerful and widely studied testing methodology, to modify the distributions of our random generators so as to find bugs with fewer tests. The key concept is combinatorial coverage, which measures the degree to which a given set of tests exercises every possible choice of values for every small combination of input features. In its “classical” form, combinatorial coverage only applies to programs whose inputs have a very particular shape—essentially, a Cartesian product of finite sets. We generalize combinatorial coverage to the richer world of algebraic data types by formalizing a class of sparse test descriptions based on regular tree expressions. This new definition of coverage inspires a novel combinatorial thinning algorithm for improving the coverage of random test generators, requiring many fewer tests to catch bugs. We evaluate this algorithm on two case studies, a typed evaluator for System F terms and a Haskell compiler, showing significant improvements in both.


Property-based testing

Combinatorial testing

Regular tree expressions

Algebraic data types

Combinatorial coverage


Harrison Goldstein

University of Pennsylvania

John Hughes

Chalmers, Data- och informationsteknik, Funktionell programmering

Quviq Ab

L. Lampropoulos

University of Maryland

Benjamin C. Pierce

University of Pennsylvania

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

03029743 (ISSN) 16113349 (eISSN)

Vol. 12648 LNCS 264-291

30th European Symposium on Programming, ESOP 2021 Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2021
Luxembourg, Luxembourg,

Systematisk testning av cyberfysiska system (SyTeC)

Vetenskapsrådet (VR) (2016-06204), 2017-01-01 -- 2022-12-31.


Sannolikhetsteori och statistik

Datavetenskap (datalogi)




Mer information

Senast uppdaterat