by Tao Yue (Simula Research Laboratory), Paolo Arcaini (National Institute of Informatics, Japan) and Shaukat Ali (Simula Research Laboratory)
Quantum software testing provides systematic and automated ways to test quantum programs to guarantee their correctness and dependability. Such a guarantee is critical to delivering the pledged revolutionary quantum computing applications to the world.
Quantum computing (QC) is a radically new computation paradigm, which lays its foundation on quantum mechanics. QC promises to deliver a revolutionary leap in computation by solving complex problems that classical computers, including powerful supercomputers, cannot handle. QC is pledging leading-edge applications, such as rapidly developing vaccines and drugs, and better understanding and predictions for cancer and other types of diseases. Developing such applications requires quantum programming. Nowadays, quantum programmers have access to a range of quantum programming languages, platforms, and simulators. However, the big challenge is to cost-effectively develop high-quality quantum software that delivers the promised revolutionary applications.
Like classical software engineering, quantum software engineering (QSE) will provide principles, processes, methods, and tools to support the design, development, maintenance, and testing of quantum software, i.e., the typical phases of a software development cycle. Since QSE is relatively new, there is currently no such life cycle for systematically developing quantum software. However, due to the availability of quantum programming languages and platforms, developing and executing quantum programs (on simulators and, to a limited extent, on quantum computers) has become possible. Furthermore, with the increasing volume of quantum software now available, the research community has started considering testing quantum programs to be an important phase.
Testing classical software has been an active research area for decades, thus there is a considerable body of knowledge in this area, which could potentially help with devising quantum software testing methods. However, quantum software adheres to quantum mechanics principles and has novel features, such as qubits, superposition, and entanglement, which introduce a new dimension of complexity, thereby, raising novel challenges on quantum software testing (see Figure 1). First, it is impossible to directly read and check quantum software states in superposition because measuring qubits results in the destruction of the superposition property of the program. In comparison, for classical software, we can instrument the program to read its states at any location whenever it is needed. Second, when testing a quantum program, it is difficult to precisely define the test oracle, i.e., a mechanism for determining whether the output is expected for a given input. For simple quantum programs, one could define probabilistic test oracles (indicating whether a test case passes/fails with a given statistical confidence), introducing further challenges such as selecting proper statistical tests and defining a suitable number of measurements/observations. Test oracles might not even exist for complex applications (e.g., weather forecasting, molecular design). Last, but not least, for classical software testing, there exist methodologies for eliciting (formally and informally), specifying, and modelling test oracles, or even generating test oracles with requirements as the basis. For quantum software testing, such methodologies do not exist.
Figure 1: Quantum software testing: Current status and way forward.
We see these issues as opportunities as much as challenges! The research community has already started taking on such challenges and obtained early achievements, as evidenced by the increasing number of works published in major software engineering venues (see Figure 1). One such achievement is Quito [1] [L1-L2], which tailors input and output coverage criteria from classical computing for testing quantum programs. Quito provides initial test suites that can give certain confidence in the correctness of the quantum program under test. However, the Quito approach faces scalability problems when testing quantum programs with high numbers of qubits; thus, requiring the development of test optimisation methods. To this end, QuSBT [2] [L3] was proposed, which employs a Genetic Algorithm (GA) to search for test suites with a maximum number of failing test cases. QuSBT has shown promising results when compared to a simple baseline (random search). Note that both Quito and QuSBT perform black-box testing, i.e., test cases are executed without reading the internal states of the quantum program.
Apart from test case generation, another important aspect of testing is the definition of the possible faults that could occur in reality. To this end, Muskit [3] [L4-L5] has been proposed as a mutation analysis approach for quantum programs. Based on various mutation operators defined specifically for quantum gates in quantum programs, Muskit can generate many mutated programs. Such programs can serve two purposes: (i) providing faulty programs for assessing the quality of test cases generated by various testing strategies; and (ii) guiding the generation of test cases. Finally, a recent effort was made to investigate the application of a classical combinatorial testing technique to quantum programs [L6].
Despite these early achievements, some challenges remain unaddressed or are insufficiently addressed in quantum software testing (see Figure 1). First, we need sophisticated solutions to measure the internal states of quantum programs to define white-box testing methods. Recent advances in projections and statistical assertions are a good start along this research line. Second, we need to develop mechanisms to define or infer test oracles for quantum programs. Metamorphic testing might be the right direction to pursue learning from classical software, which has shown promising results for testing programs for which test oracles do not exist. Third, we need intuitive testing solutions readily usable by test engineers. Currently, the development and testing of quantum programs are at the low-level quantum circuits. Last, testing techniques are currently developed for noiseless simulators, but the generated tests must be ultimately executed on physical hardware subject to noise, which is still unavoidable in the current and near-future quantum computers. However, significant effort is being made to eliminate the noise from QC hardware.
Links:
[L1] https://github.com/Simula-COMPLEX/quito
[L2] https://www.youtube.com/watch?v=kuI9QaCo8A8
[L3] https://github.com/Simula-COMPLEX/qusbt
[L4] https://github.com/Simula-COMPLEX/muskit
[L5] https://www.youtube.com/watch?v=EbPHJOK_AEA
[L6] https://github.com/Simula-COMPLEX/qucat-paper
References:
[1] S. Ali, P. Arcaini, X. Wang and T. Yue: “Assessing the Effectiveness of Input and Output Coverage Criteria for Testing Quantum Programs”, 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), 2021, pp. 13-23, doi: 10.1109/ICST49551.2021.00014.
[2] X. Wang, P. Arcaini, T. Yue, S. Ali: “Generating Failing Test Suites for Quantum Programs With Search”, in: O'Reilly UM., Devroey X. (eds) Search-Based Software Engineering, SSBSE 2021, LNCS, vol 12914, Springer, https://doi.org/10.1007/978-3-030-88106-1_2
[3] E. Mendiluze, S. Ali, P. Arcaini, and T. Yue: “Muskit: A mutation analysis tool for quantum software testing”, in The 36th IEEE/ACM Int. Conf. on Automated Software Engineering, Tool Demonstration, IEEE/ACM, 2021.
Please contact:
Tao Yue
Simula Research Laboratory, Norway
