by Nicola Ferro
Measuring is a key to scientific progress. This is particularly true for research concerning complex systems, whether natural or human-built. PROMISE will provide a virtual laboratory for conducting participative research and experimentation to carry out, advance and bring automation into the evaluation and benchmarking of complex multilingual and multimedia information systems.
With a population of over 500 million in its 27 states in which more than 80 indigenous and many more immigrant languages are found, the citizens and companies of the EU demand information systems that allow them to interact with the culturally and politically diverse content that surrounds them in multiple media. Moreover, with the advance of broadband access and the evolution of both wired and wireless connection modes, users are now not only information consumers, but also information producers: language and media barriers are no longer seen as inviolable and they are constantly crossed and mixed to provide content that can be accessed on a global scale within a multicultural and multilingual setting.
The technology and research behind multilingual and multimedia information systems are, today, in the position of intercepting these emerging trends but their design and development is becoming increasingly complex and needs proper means for ensuring that they meet the expected user requirements and provide the desired effectiveness.
Objectives
We consider experimental evaluation – both laboratory and interactive – a key means for supporting and fostering the development of multilingual and multimedia information systems in order to ensure that they meet new and evolving user requirements, provide the desired effectiveness and efficiency, guarantee the required robustness and reliability, and operate with the necessary scalability.
PROMISE (Participative Research labOratory for Multimedia and Multilingual Information Systems Evaluation) aims at advancing the experimental evaluation of complex multimedia and multilingual information systems in order to support individuals, commercial entities, and communities who design, develop, employ, and improve such complex systems. The overall goal of PROMISE is to deliver a unified environment collecting data, knowledge, tools, methodologies, and the user community which are involved in the experimental evaluation.
Figure 1: Activities of PROMISE.
PROMISE is a network of excellence (contract n. 258191), funded in the 7th Framework Programme of the European Commission, with 10 academic and industrial partners, who are leaders in the field: University of Padua, Italy (coordinator); SICS, Sweden; ; University of Amsterdam, The Netherlands; Sapienza University of Rome, Italy; University of Applied Sciences Western Switzerland, Switzerland; Information Retrieval Facility (IRF), Austria; Zurich University of Applied Sciences, Switzerland; Humboldt-Universität zu Berlin, Germany; Evaluations and Language resources Distribution Agency (ELDA), France; Centre for the Evaluation of Language Communication Technologies (CELCT), Italy. The PROMISE project lasts three years, starting from September 2010 and ending in August 2013.
Actvities
To achieve its goals, PROMISE will pursue the following activites:
- Foster the adoption of regular and thorough experimental evaluation activities: PROMISE will carry on the successful and renowned CLEF (Cross-Language Evaluation Forum) evaluation campagins futher pushing the evaluation exercises to tackle realistic tasks, use cases, and data sets;
- Bring automation into the experimental evaluation process: PROMISE will propose methods and provide software infrastructure to create larger experimental collections; increase the number and size of the experiments conducted; and develop distributed, asynchronous, and loosely-coupled evaluation protocols, moving experimental evaluation from handcraft process to a mostly automatic one.
- Promote collaboration and re-use over the acquired knowledge-base: PROMISE will curate, preserve, and enrich the collected experimental data; provide the means for an easy comparison with and a meaningful interpretation and visualisation of the experimental results; and facilitate the discussion and collaboration among all the interested stakeholders.
- Stimulate knowledge transfer and uptake: PROMISE will disseminate know-how, tools, and best practices about multilingual and multimedia information systems; facilitate uptake and participation by commercial entities and industries; and give rise to multidisciplinary competencies and expertises.
Links:
PROMISE: http://www.promise-noe.eu/
CLEF 2010: http://www.clef2010.org/
CLEF 2011: http://www.clef2011.org/
CLEF: http://www.clef-campaign.org/
Please contact:
Nicola Ferro, University of Padua, Italy
E-mail:
