Language-Based Techniques and Stochastic Models for Automated Testing
Doktorsavhandling, 2023

As software systems become bigger and scarier, automating their testing is crucial to ensure that our confidence in them can keep up with their growth. In this setting, Generational Fuzzing and Random Property-Based Testing are two sides of the same testing technique that can help us find bugs effectively without having to spend countless hours writing unit tests by hand. They both rely on generating large amounts of random (possibly broken) test cases to be used as inputs to the system. Test cases that trigger issues such as crashes, memory leaks, or failed assertions are reported back to the developer for further investigation. Despite being fairly automatable, the Achilles heel of this technique lies in the quality of the randomly generated test cases, often requiring substantial manual work to tune the random generation process when the system under test expects inputs satisfying complex invariants.

This thesis tackles this problem from the Programming Languages perspective, taking advantage of the richness of functional, statically-typed languages like Haskell to develop automated techniques for generating good-quality random test cases, as well as for automatically tuning the testing process in our favor. To this purpose, we rely on well-established ideas such as coverage-guided fuzzing, meta-programming, type-level programming, as well as novel interpretations of centuries-old statistical tools designed to study the evolution of populations such as branching processes. All these ideas are empirically validated using an extensive array of case studies and supported by a substantial number of real-world bugs discovered along the way.

automated testing

software testing

stochastic methods

meta-programming

functional programming

Edit room EB
Opponent: Kostis Sagonas, Uppsala University, Sweden

Författare

Claudio Agustin Mista

Chalmers, Data- och informationsteknik, Informationssäkerhet

Branching Processes for QuickCheck Generators

Haskell 2018 - Proceedings of the 11th ACM SIGPLAN International Symposium on Haskell, co-located with ICFP 2018,;Vol. 53(2018)p. 1-13

Paper i proceeding

Deriving Compositional Random Generators

ACM International Conference Proceeding Series,;Vol. 25 September 2019(2019)

Paper i proceeding

Generating Random Structurally Rich Algebraic Data Type Values

Proceedings - 2019 IEEE/ACM 14th International Workshop on Automation of Software Test, AST 2019,;(2019)p. 48-54

Paper i proceeding

Short Paper: Weak Runtime-Irrelevant Typing for Security

PLAS 2020 - Proceedings of the 15th Workshop on Programming Languages and Analysis for Security,;(2020)p. 13-17

Paper i proceeding

BinderAnn: Automated Reification of Source Annotations for Monadic EDSLs

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics),;Vol. 12222 LNCS(2020)p. 25-46

Paper i proceeding

MUTAGEN: Reliable Coverage-Guided, Property-Based Testing using Exhaustive Mutations

Proceedings - 2023 IEEE 16th International Conference on Software Testing, Verification and Validation, ICST 2023,;(2023)p. 176-187

Paper i proceeding

QuickFuzz testing for fun and profit

Journal of Systems and Software,;Vol. 134(2017)p. 340-354

Artikel i vetenskaplig tidskrift

Modern software systems are incredibly complex, so we need clever ways to test them automatically to avoid losing our minds. Generational Fuzzing and Random Property-Based Testing are two popular testing approaches that can be fairly automated. They rely on generating loads of random (and occasionally broken) test cases to uncover bugs while saving us from writing endless unit tests. But there is a catch: generating good random test cases automatically is not easy. Complex software systems demand inputs that satisfy tricky rules, making automated random generation a challenge.

This thesis tackles this conundrum using the features present in functional programming languages with strong guarantees like Haskell. In particular, we develop automated techniques to produce effective test cases and fine-tune the testing process. Our ideas are based on techniques like coverage-guided fuzzing, meta- and type-level programming, and we even borrow some statistical tricks from centuries-old population studies! We put our methods to the test through an assorted range of case studies, discovering several real-world bugs along the way.

Octopi: Säker Programering för Sakernas Internet

Stiftelsen för Strategisk forskning (SSF) (RIT17-0023), 2018-03-01 -- 2023-02-28.

WebSec: Säkerhet i webb-drivna system

Stiftelsen för Strategisk forskning (SSF) (RIT17-0011), 2018-03-01 -- 2023-02-28.

Styrkeområden

Informations- och kommunikationsteknik

Ämneskategorier

Datorsystem

ISBN

978-91-7905-851-7

Doktorsavhandlingar vid Chalmers tekniska högskola. Ny serie: 5317

Utgivare

Chalmers

Edit room EB

Online

Opponent: Kostis Sagonas, Uppsala University, Sweden

Mer information

Senast uppdaterat

2023-06-15