In recent years, software quality has begun to gain great importance, principally because of the key role software plays in our day-to-day lives. To control software quality, it is necessary to conduct evaluations of the software products themselves. The AQC Lab was established for this purpose and its core responsibility is evaluating the quality of software products against the ISO/IEC 25000 standard.
Although numerous certifications for software quality exist (e.g., ISO/IEC15504, CMMI, etc.), there is little evidence to suggest that compliance with any of these standards guarantees good software products. Critics have gone so far as to suggest that the only thing these standards guarantee is uniformity of output and thus, may actually lead to the production of bad products. Consequently, the idea that software evaluations should be based on direct evidence of product attributes is becoming more widespread. Therefore, a growing number of organizations are becoming concerned about the quality of the products that they develop and/or acquire, as well as the processes.
The ISO/IEC 25000 family of standards, known as SQuaRE (Software Product Quality Requirements and Evaluation), appears to meet this emerging need to assess product-related quality. The objective of ISO/IEC 25000 is to create a common framework within which to evaluate software product quality and this standard is now beginning to replace the previous ISO/IEC 9126 and ISO/IEC 14598 standards to become the cornerstone of this area of software engineering. ISO/IEC25000 is divided into several parts: we highlight ISO/IEC25040 [1] which defines the process of evaluating software product quality and ISO/IEC25010 [2] which determines the software product characteristics and sub-characteristics that can be evaluated.
As with other standards, ISO/IEC 25000 describes what to evaluate but does not specify how. In other words, it does not detail the thresholds for the evaluation metrics to be used, nor does it describe how to group these metrics in order to assign a quality value to a software product.
Improving this evaluation process has been the goal for AQC team (began by the Alarcos Research Group, University of Castilla-La Mancha) over the last few years. This has led to the creation of AQC Lab [3], the first laboratory accredited for ISO/ IEC17025 by ENAC (National Accreditation Entity) to assess the quality of software products using the ISO/IEC25000. The lab has also been recognised by the ILAC (International Laboratory Accreditation Cooperation). This accreditation confirms the technical competence of the laboratory and ensures the reliability of the evaluation results. The AQC Lab uses three main elements to conduct the quality evaluations. These are:
- The Assessment Process which directly adopts the activities of ISO/IEC 25040 and completes them with specific laboratory roles and instructions which have been developed internally. This process produces an evaluation report that shows the level of quality achieved by the product and any software aspects that may require improvement.
- The Quality Model which defines the characteristics and metrics needed to evaluate the software product. This model was developed through the MEDUSAS (Improvement and Evaluation of Usability, Security and Maintainability of Software (2009-2012)) research project, funded by MICINN/ FEDER. Although the AQC Lab evaluates multiple features of the model presented in ISO/IEC 25010, the accreditation initially focuses on the characteristic of maintainability, principally because maintenance is one of the most expensive phases of the development lifecycle and maintainability is one of the features most frequently requested by software clients, Many clients seek products that maintain themselves or can be maintained by a third party. This model required considerable effort to develop, eventually resulting in a set of quality properties that are measurable from the source code and related to the sub-characteristics of quality proposed in ISO/IEC25010 (Table 1).
- The Evaluation Environment which largely automates the evaluation tasks. This environment uses measurement tools that are applied in the software product to combine the values obtained, assign quality levels to the model sub-characteristics and characteristics and, eventually, present them in an easily accessible manner.
Table 1: Relationship between the sub-characteristics and quality properties in the quality model (completed in Step 2 of the evaluation process).
In addition, the Spanish Association for Standardization and Certification (AENOR) has created a software product quality certification based on ISO/ IEC25000. To perform a certification, AENOR reviews the assessment report issued by the accredited laboratory and makes a brief audit of the company and the product. If everything is correct, the company is then given a certificate of quality for its software product.
During the past year, several pilot projects have been run using this certification scheme. Participating companies (from Spain and Italy) have been the first to be assessed and future projects are planned for companies in Colombia, Mexico and Ecuador.
Links:
http://iso25000.com/
http://www.alarcosqualitycenter.com/index.php/software-quality-evaluation-lab
References:
[1] ISO, ISO/IEC 25040 Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - Evaluation process. 2011: Ginebra, Suiza.
[2] ISO, ISO/IEC 25010, Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models. 2011: Ginebra, Suiza.
[3] Verdugo, J., M. Rodríguez, and M. Piattinni, Using Agile Methods to Implement a Laboratory for Software Product Quality Evaluation, in 15th International Conference on Agile Software Development (XP 2014). 2014: Roma (Italia).
Please contact:
Moisés Rodríguez
Alarcos Quality Center, Spain
E-mail:
Mario Piattini
University of Castilla–La Mancha, Spain
E-mail:
