Language-Based Techniques and Stochastic Models for Automated Testing
Doctoral thesis, 2023

As software systems become bigger and scarier, automating their testing is crucial to ensure that our confidence in them can keep up with their growth. In this setting, Generational Fuzzing and Random Property-Based Testing are two sides of the same testing technique that can help us find bugs effectively without having to spend countless hours writing unit tests by hand. They both rely on generating large amounts of random (possibly broken) test cases to be used as inputs to the system. Test cases that trigger issues such as crashes, memory leaks, or failed assertions are reported back to the developer for further investigation. Despite being fairly automatable, the Achilles heel of this technique lies in the quality of the randomly generated test cases, often requiring substantial manual work to tune the random generation process when the system under test expects inputs satisfying complex invariants.

This thesis tackles this problem from the Programming Languages perspective, taking advantage of the richness of functional, statically-typed languages like Haskell to develop automated techniques for generating good-quality random test cases, as well as for automatically tuning the testing process in our favor. To this purpose, we rely on well-established ideas such as coverage-guided fuzzing, meta-programming, type-level programming, as well as novel interpretations of centuries-old statistical tools designed to study the evolution of populations such as branching processes. All these ideas are empirically validated using an extensive array of case studies and supported by a substantial number of real-world bugs discovered along the way.

software testing


stochastic methods

functional programming

automated testing

Edit room EB
Opponent: Kostis Sagonas, Uppsala University, Sweden


Claudio Agustin Mista

Chalmers, Computer Science and Engineering (Chalmers), Information Security

Branching Processes for QuickCheck Generators

Haskell 2018 - Proceedings of the 11th ACM SIGPLAN International Symposium on Haskell, co-located with ICFP 2018,; Vol. 53(2018)p. 1-13

Paper in proceeding

Deriving Compositional Random Generators

ACM International Conference Proceeding Series,; Vol. 25 September 2019(2019)

Paper in proceeding

Generating Random Structurally Rich Algebraic Data Type Values

Proceedings - 2019 IEEE/ACM 14th International Workshop on Automation of Software Test, AST 2019,; (2019)p. 48-54

Paper in proceeding

Short Paper: Weak Runtime-Irrelevant Typing for Security

PLAS 2020 - Proceedings of the 15th Workshop on Programming Languages and Analysis for Security,; (2020)p. 13-17

Paper in proceeding

BinderAnn: Automated Reification of Source Annotations for Monadic EDSLs

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics),; Vol. 12222 LNCS(2020)p. 25-46

Paper in proceeding

MUTAGEN: Reliable Coverage-Guided, Property-Based Testing using Exhaustive Mutations

Proceedings - 2023 IEEE International Conference on Software Testing, Verification and Validation, ICST 2023,; (2023)

Paper in proceeding

QuickFuzz testing for fun and profit

Journal of Systems and Software,; Vol. 134(2017)p. 340-354

Journal article

Modern software systems are incredibly complex, so we need clever ways to test them automatically to avoid losing our minds. Generational Fuzzing and Random Property-Based Testing are two popular testing approaches that can be fairly automated. They rely on generating loads of random (and occasionally broken) test cases to uncover bugs while saving us from writing endless unit tests. But there is a catch: generating good random test cases automatically is not easy. Complex software systems demand inputs that satisfy tricky rules, making automated random generation a challenge.

This thesis tackles this conundrum using the features present in functional programming languages with strong guarantees like Haskell. In particular, we develop automated techniques to produce effective test cases and fine-tune the testing process. Our ideas are based on techniques like coverage-guided fuzzing, meta- and type-level programming, and we even borrow some statistical tricks from centuries-old population studies! We put our methods to the test through an assorted range of case studies, discovering several real-world bugs along the way.

Octopi: Säker Programering för Sakernas Internet

Swedish Foundation for Strategic Research (SSF) (RIT17-0023), 2018-03-01 -- 2023-02-28.

WebSec: Securing Web-driven Systems

Swedish Foundation for Strategic Research (SSF) (RIT17-0011), 2018-03-01 -- 2023-02-28.

Areas of Advance

Information and Communication Technology

Subject Categories

Computer Systems



Doktorsavhandlingar vid Chalmers tekniska högskola. Ny serie: 5317



Edit room EB


Opponent: Kostis Sagonas, Uppsala University, Sweden

More information

Latest update