Automating Black-Box Property Based Testing

Doctoral thesis, 2016

Black-box property based testing tools like QuickCheck allow developers to write elegant logical specifications of their programs, while still permitting unrestricted use of the same language features and libraries that simplify writing the programs themselves. This is an improvement over unit testing because a single property can replace a large collection of test cases, and over more heavy-weight white-box testing frameworks that impose restrictions on how properties and tested code are written. In most cases the developer only needs to write a function returning a boolean, something any developer is capable of without additional training. This thesis aims to further lower the threshold for using property based testing by automating some problematic areas, most notably generating test data for user defined data types. Writing procedures for random test data generation by hand is time consuming and error prone, and most fully automatic algorithms give very poor random distributions for practical cases. Several fully automatic algorithms for generating test data are presented in this thesis, along with implementations as Haskell libraries. These algorithms all fit nicely within a framework called sized functors, allowing re-usable generator definitions to be constructed automatically or by hand using a few simple combinators. Test quality is another difficulty with property based testing. When a property fails to find a counterexample there is always some uncertainty in the strength of the property as a specification. To address this problem we introduce a black-box variant of mutation testing. Usually mutation testing involves automatically introducing errors (mutations) in the source code of a tested program to see if a test suite can detect it. Using higher order functions, we mutate functions without accessing their source code. The result is a very light-weight mutation testing procedure that automatically estimates property strength for QuickCheck properties.