by Stefano Chessa, Francesco Furfari, Francesco Potortì, Juan Pablo Lázaro and Dario Salvi
Owing to the complexity of Ambient Assisted Living (AAL) systems and platforms, the evaluation of AAL solutions is a complex task that will challenge researchers for years to come. However, the analysis and comparison of proposed solutions is paramount to enable us to assess research results in this area. We have thus organized an international contest called EvAAL: Evaluating AAL Systems Through Competitive Benchmarking. Its aims are to raise interest within the research and developer communities in the multidisciplinary research fields enabling AAL, and to create benchmarks for the evaluation and comparison of AAL systems.
The EvAAL competition is the first project incubated by the AALOA community. It aims at establishing benchmarks and evaluation metrics to compare different AAL solutions and assess advances in the field. In recognition of the complexity of AAL systems, in the early editions of EvAAL only AAL components will be compared. Later, as methodologies and tools of EvAAL become more mature, aggregates of components, services and even complete systems will be evaluated.
EvAAL is organized as a multi-track conference, with each track covering different aspects, components or services of AAL systems. An annual call for ideas solicits track proposals. For each topic selected by the EvAAL steering board, a technical committee writes the call for competition and the rules (including benchmarks and evaluation metrics), and selects the competing teams from the applicants responding to the call for competition.
EvAAL will be run each year in a host living lab offering an environment that simulates every-day life situations. The participating teams are provided with a kit that includes available information about the living lab (including a map and interfaces to access its facilities such as cameras, automatisms etc.), the evaluation criteria and the evaluation software. Competitors are also supported in the adaptation/preparation of their artifacts by the technical committee.
The competition lasts several days with a time slot assigned to each competing team. The results are communicated at an award ceremony in a final workshop in which the status of the experiments is reported.
The first edition of EvAAL was run at the CIAMI living lab in Valencia in July 2011 with a single track entitled “Indoor Localization and Tracking”. These topics were chosen because they comprise a very common building block of AAL systems that need to provide their services at the right time in the right place to their users. Seven teams (selected from ten applicants) from six countries (Austria, France, Germany, Spain, Switzerland, Ukraine) participated. The technologies and methodologies covered by the competing teams were extremely heterogeneous, including infra-red, ultra-sound, radio signal strength based on Wi-Fi and ZigBee and radio frequency phase measurements, and capacitive carpets.
Figure 1: A picture of the CIAMI living lab taken from a camera on the ceiling.
During the competition, each system had to track the position of the user (impersonated by an actor) moving around the lab along pre-defined paths and standing in predefined spaces. Figure 1 shows a picture taken from a ceiling camera in the living lab. The red and blue signs on the ground mark the right and left steps along the predefined paths. Each system was subject to two benchmarks: Area of interest and tracking. In the first test, the system had to detect the presence of the user in five different square areas, with 50 cm sides (the areas of interest). In the tracking test, each system had to track the user along five different predefined paths. For each path, systems had to localize the user in real time by providing an updated position every half second. The paths and the area of interest simulated movements corresponding to specific activities of the user (eg cooking, waking up, etc.). In addition to the accuracy of localization information, four other metrics were used to compare the systems: installation complexity (measured as total number of minutes/persons required for the installation of the artifacts), availability (measured as the fraction of available localization data over the number of expected data), user acceptance (a grade given by the members of the evaluation committee), and integrability into AAL systems (a score dependent on the availability of libraries and tools for the integration of the artifact and on the use of standards and open source solutions).
The results of the competition were presented at the EvAAL workshop, an AAL Forum satellite workshop held in Lecce, Italy (26-28 Sept 2011).
We invite all those interested to participate in the process of identification of new topics and ideas (beyond localization and tracking) for the forthcoming editions.
Link: http://evaal.aaloa.org
Please contact:
Stefano Chessa
EvAAL Steering Board Chair
ISTI-CNR and Pisa University, Italy
E-mail:
