On Optimization-Based Falsification of Cyber-Physical Systems
Doctoral thesis, 2022
In what is commonly referred to as cyber-physical systems (CPSs), computational and physical resources are closely interconnected. An example is the closed-loop behavior of perception, planning, and control algorithms, executing on a computer and interacting with a physical environment. Many CPSs are safety-critical, and it is thus important to guarantee that they behave according to given specifications that define the correct behavior. CPS models typically include differential equations, state machines, and code written in general-purpose programming languages. This heterogeneity makes it generally not feasible to use analytical methods to evaluate the system’s correctness. Instead, model-based testing of a simulation of the system is more viable.
Optimization-based falsification is an approach to, using a simulation model, automatically check for the existence of input signals that make the CPS violate given specifications. Quantitative semantics estimate how far the specification is from being violated for a given scenario. The decision variables in the optimization problems are parameters that determine the type and shape of generated input signals.
This thesis contributes to the increased efficiency of optimization-based falsification in four ways. (i) A method for using multiple quantitative semantics during optimization-based falsification. (ii) A direct search approach, called line-search falsification that prioritizes extreme values, which are known to often falsify specifications, and has a good balance between exploration and exploitation of the parameter space. (iii) An adaptation of Bayesian optimization that allows for injecting prior knowledge and uses a special acquisition function for finding falsifying points rather than the global minima. (iv) An investigation of different input signal parameterizations and their coverability of the space and time and frequency domains.
The proposed methods have been implemented and evaluated on standard falsification benchmark problems. Based on these empirical studies, we show the efficiency of the proposed methods. Taken together, the proposed methods are important contributions to the falsification of CPSs and in enabling a more efficient falsification process.
Optimization-based falsification is an approach to, using a simulation model, automatically check for the existence of input signals that make the CPS violate given specifications. Quantitative semantics estimate how far the specification is from being violated for a given scenario. The decision variables in the optimization problems are parameters that determine the type and shape of generated input signals.
This thesis contributes to the increased efficiency of optimization-based falsification in four ways. (i) A method for using multiple quantitative semantics during optimization-based falsification. (ii) A direct search approach, called line-search falsification that prioritizes extreme values, which are known to often falsify specifications, and has a good balance between exploration and exploitation of the parameter space. (iii) An adaptation of Bayesian optimization that allows for injecting prior knowledge and uses a special acquisition function for finding falsifying points rather than the global minima. (iv) An investigation of different input signal parameterizations and their coverability of the space and time and frequency domains.
The proposed methods have been implemented and evaluated on standard falsification benchmark problems. Based on these empirical studies, we show the efficiency of the proposed methods. Taken together, the proposed methods are important contributions to the falsification of CPSs and in enabling a more efficient falsification process.
Bayesian Optimization
Model-Based Testing
Optimization-Based Falsification
Cyber-Physical Systems
Quantitative Semantics
Input Generators
Author
Zahra Ramezani
Chalmers, Electrical Engineering, Systems and control
Verification and testing activities are used to increase the confidence that embedded systems fulfill given requirements. The requirements can express, for example, safety, performance, or legal aspects that the system must meet.
Increased computational capabilities, together with more advanced sensors, make it possible for computer-controlled systems to take over many activities previously done by humans. However, the increased complexity of using algorithms to handle perception, decision making, and actuation means that there is a strong industrial need for new methods that can support the design and development phases of new systems so they are guaranteed to fulfill their requirements.
This thesis concerns methods and tools for assisting the development of, for example, safety-critical autonomous systems. In particular, we consider the
falsification problem, which tries to find out if there exist situations in which the given requirements are not fulfilled. If such a situation is found, we have a counterexample that can be used by the engineers to update the system to not break the requirements.
Specifically, in this thesis, we investigate how optimization methods can be used to find these counterexamples with as few simulations of the system as possible. By evaluating the proposed methods on many benchmark problems we are able to find more counterexamples faster than with previous methods.
Increased computational capabilities, together with more advanced sensors, make it possible for computer-controlled systems to take over many activities previously done by humans. However, the increased complexity of using algorithms to handle perception, decision making, and actuation means that there is a strong industrial need for new methods that can support the design and development phases of new systems so they are guaranteed to fulfill their requirements.
This thesis concerns methods and tools for assisting the development of, for example, safety-critical autonomous systems. In particular, we consider the
falsification problem, which tries to find out if there exist situations in which the given requirements are not fulfilled. If such a situation is found, we have a counterexample that can be used by the engineers to update the system to not break the requirements.
Specifically, in this thesis, we investigate how optimization methods can be used to find these counterexamples with as few simulations of the system as possible. By evaluating the proposed methods on many benchmark problems we are able to find more counterexamples faster than with previous methods.
Subject Categories
Control Engineering
ISBN
978-91-7905-732-9
Doktorsavhandlingar vid Chalmers tekniska högskola. Ny serie: 5198
Publisher
Chalmers