Software Robustness: From Requirements to Verification

Doctoral thesis, 2013

The importance of software quality increases as software products become more intertwined with our everyday lives. A critical software quality attribute is robustness, i.e. that the software shows stable behavior in stressful conditions and when receiving faulty inputs. Even though this has been a long-term goal in software engineering, few studies directly target robustness. The overall goal of this thesis is to identify gaps in the knowledge and take steps towards improving and creating methods to work with software robustness. To identify gaps in the state of knowledge, this thesis first describes a systematic review of the academic literature on software robustness. The results, based on analysis of 144 relevant papers, suggest that the most prominent contributions on robustness are methods and tools for random testing on the external interfaces of systems. Another finding is the lack of empirical evidence and guidelines on how to dene and specify robustness. Additionally, there is a lack of methods to elicit, analyze, and specify robustness requirements in a systematic way, and to test these requirements. To address the goals of the thesis, we have worked with ve industrial companies. We examined the state of practice by conducting interviews and analyzing requirements documents at some of our partner companies to identify improvement potential. The results show that there also is a lack of systematic methods to specify and test quality requirements in practice. Furthermore, unverifiable quality requirements are still a source of problem and high cost to software development projects. To address these issues, we constructed a framework for analysis, elicitation, and specification of software robustness (ROAST). Based on simple models for root causes and symptoms of robustness failures, we have identified 19 patterns for robustness requirements. Further, ROAST includes a notion of specification levels that helps practitioners refine high-level requirements to a verifiable level. The framework has been evaluated using document analysis, interviews, and surveys at the partner companies. The evaluations have investigated the usefulness, quality, and generalizability of ROAST and have helped us improve the framework over time. The last part of the thesis uses the patterns in ROAST, to specify generic robustness properties that the system should fulfill. We present a testing framework, RobusTest, that uses these properties to automatically generate robustness test cases. This provides a more focused testing than complete random testing. We have implemented and evaluated parts of this framework and found robustness issues in open source and well-tested industrial systems. This thesis provides guidelines for and discusses how practitioners can more systematically work with robustness from requirements elicitation and analysis to testing.