by Tanja E. J. Vos
Complex systems exhibit emergent behaviour, which makes them hard to predict. This presents particularly challenging problems during testing. However, this is a challenge that simply cannot be avoided: testing is a vital part of the quality assurance process. With important partners from Spain (ITI), UK (Kings College), France (INRIA), Germany (Fraunhofer, Daimler, BMS), Bulgaria (RILA) and Greece (European Dynamics), the EvoTest project, funded under the 6th Framework programme (IST-33742), is attacking the problem of testing complex systems using evolutionary algorithms.
The EvoTest project addresses a fundamental problem faced by the European software industry: quality assurance of complex software systems. There is strong empirical evidence that deficient testing of both functional and non-functional properties is one of the major sources of software and system errors. Even though many test automation tools are currently available to aid test planning and control as well as test case execution and monitoring, all these tools share a similar passive philosophy towards test case design, selection of test data and test evaluation. They leave these crucial, time-consuming and demanding activities to the human tester. This is not without reason; test case design and test evaluation are difficult to automate with the techniques available in current industrial practice. The domain of possible inputs (potential test cases), even for a trivial program, is typically too large to be exhaustively explored. One of the major challenges associated with test case design is the selection of test cases that are effective at finding flaws without requiring an excessive number of tests to be carried out. This is the problem that EvoTest has taken on.
The strategy is to combine various visions of the IST programme: a multidisciplinary approach to evolving, adaptive and automated testing as a solution to some of the challenges of mastering complexity in software development. During its running period, 2006-2009, the project has successfully developed, applied and evaluated Evolutionary Testing techniques. Evolutionary Testing is an exciting, novel, nature-inspired solution, which transforms testing into an optimization problem. This allows the problem to be attacked using search techniques inspired by Darwinian evolution. To increase the test efficiency for complex systems EvoTest has developed an Automated Evolutionary Testing Framework. This framework is able to automatically search for high-quality test cases with a high probability of error detection. Automation is widely regarded as the key to efficient testing.
The impossibility of anticipating or testing all possible uses and behaviours of a (complex) system suggests a prominent role for Evolutionary Testing, because it relies on very few assumptions about the underlying problem it is attempting to solve. In addition, optimization and search techniques are adaptive and, therefore, able to modify their behaviour when faced with unforeseen situations. These two properties make evolutionary testing approaches ideal for handling complex systems.
EvoTest has reached the following objectives:
- An extensible and open Automated Evolutionary Testing Framework has been developed. The framework provides general components and interfaces to facilitate the automatic generation, execution, monitoring and evaluation of black-box and white-box testing of C programs.
- The performance and the quality of the Evolutionary White-box Testing has been increased by combining evolutionary testing with other software engineering techniques, such as slicing and program transformation. These advanced techniques have been tailored both to reduce the size of the search space and to transform the search space.
- The EvoTest tools and techniques have been evaluated using real-world case studies from the automotive and telecommunications industries. Case studies have been performed for black-box and white-box testing activities.
EvoTest results have made it abundantly clear that evolutionary testing holds great promise as a solution to many challenges faced by the software industry. Future work on Evolutionary Testing should concentrate on theoretical foundations, search technique improvements, new testing objectives, tool environment/testing infrastructure, and new application areas.
The EvoTest project has brought together European entities that are world leaders in this emerging field. ITI (Spain) is project coordinator and research partner providing expertise in software testing and evolutionary computation. Moreover, ITI acts as software developer, implementing various components of the Framework. INRIA (France) serves as research partner providing expertise in evolutionary computation. Fraunhofer FIRST (Germany) is a research partner providing expertise in software development, technology, validation and verification. Kings College (United Kingdom) is a research partner providing expertise in software engineering techniques like slicing and program transformations. Daimler (Germany) is a case study provider of a complex automotive application. Moreover, they serve as research partner contributing expertise on evolutionary testing. Berner & Mattner (Germany) is a case study provider of testing infrastructure. Rila (Bulgaria) is a case study provider of a complex accessibility solution, and acts as a software developer. European Dynamics (Greece) is a technology provider and software developer.
Link:
http://www.evotest.eu
Please contact:
Tanja E.J. Vos
Universidad Politécnica de Valencia/SpaRCIM
Tel: +34 690 917 971
E-mail: tvos
pros.upv.es
