BaseIT -- Basing Software Testing on Information Theory

Research Project, 2016 – 2019

Software is a critical part of our modern society but it has been notoriously hard to create a mathematically grounded basis for either the software itself or its development. In previous research we have argued that information theory is a natural area with which to build such a foundation. In this project (BaseIT -- Basing Software Testing on Information Theory) we will create such a scientific basis for software testing by grounding it in information theory. In much the same way that information theory has strengthened the area of cryptography and lead to cryptographic systems that are information theoretically secure this project will strengthen the area of software testing and lead to testing methods that are information theoretically optimal. Additionally, we will also investigate to what extent the developed basis can be extended to software engineering in general. The project has five, main activities, two of which focus on two specific software testing problems while three develop explicit methodologies for applying information theory in software testing. Focus area 1 extends the previous universal test diversity metrics proposed by the PI to apply to sets of test cases rather than only to pairs. This can be leveraged to study the diversity of groups or whole suites of test cases as well as optimize this diversity in automated testing tools. Together this helps make the intuitive notion that "Test cases in a test suite should be as diverse as possible" well-defined and quantifiable, thus transforming it from a vague, subjective statement to a precise one which can be studied and exploited. Similarly, focus area 2 investigates the classic, but ad hoc and vague testing guideline of Boundary Value Analysis that "Test inputs should be selected on both sides of boundaries of the input domain". In both focus areas we utilize recent results from information theory of how to quantify the information distance between general objects. In addition to being theoretically motivated this gives practical benefits for testing tools since they apply regardless of data types of inputs/outputs, programming languages or technologies the software under test is based on. By coupling the developed measures to search-based software testing technologies practical testing tools will result. The focus areas provide concrete examples of powerful information theoretical analysis and results and the benefits they provide in software testing. But the focus areas also provide input to develop general methodologies for how to apply information theoretical analysis in software testing. The methdologies are built by three further activities in the BaseIT project. The first one assembles the existing but scarce literature within software engineering and software testing that applies information theoretical concepts and analyzes which approaches and information sources have been used. This is input to the second and third methodology activities of the project. The former focuses on the analytical strategies from information theory that can be applied in software testing and describes a methodology for how to select and apply them. The latter focuses on information sources to be used and the cost of collecting and maintaining the information they provide. This will be used in a methodology to trade-off the cost of a software testing technique with its benefits and, as such, to help increase the likelihood for practically applicable software testing results. A number of ongoing, parallel projects can be further strengthened by and strengthen BaseIT as can existing international collaborations. The previous experience of the PI in applying information theoretical ideas to software testing problems as well as being one of the early pioneers of search-based software engineering makes it highly likely that both methodological