by Nicola Ferro
Since 2000, CLEF has played a successful role in stimulating research and promoting evaluation in a wide range of key areas in the information access and retrieval domain. In 2010, a radical innovation and renewal process led to the establishment of the CLEF Initiative, whose mission is to promote research, innovation, and development of information access systems with emphasis on multilinguality and multimodality.
The CLEF Initiative is structured in two main parts:
1. a series of Evaluation Labs, i.e., laboratories to conduct evaluation of information access systems and workshops to discuss and pilot innovative evaluation activities;
2. a peer-reviewed Conference on a broad range of issues, including
- the activities of the Evaluation Labs;
- experiments using multilingual and multimodal data; in particular, but not only, data resulting from CLEF activities;
- research in evaluation methodologies and challenges.
Due to these changes and the broader scope of the CLEF Initiative, the acronym CLEF, traditionally expanded to Cross-Language Evaluation Forum, now translates to Conference and Labs of the Evaluation Forum. This renewal process and the organization of the annual CLEF events are partially supported by the EU FP7 PROMISE project (contract n. 258191): Participative Research labOratory for Multimedia and Multilingual Information Systems Evaluation and by the ELIAS RNP network.
CLEF 2012 participants.
CLEF 2012: Information Access Evaluation meets Multilinguality, Multimodality, and Visual Analytics
The annual meeting of the CLEF Initiative was hosted this year by the Sapienza University of Rome, Italy, 17-20 September as a 4 day event in which conference presentations, laboratory meetings, workshops, and community sessions were smoothly interleaved to provide a continuous stream of discussions on the different facets of experimental evaluation.
14 papers and 3 posters were accepted for the Conference and published by Springer in their Lectures Notes for Computer Science (LNCS) series. Two keynote speakers highlighted important developments in the field of evaluation. Peter Clark, Vulcan Inc., USA, focused on how to move from information retrieval to knowledgeable machines. Tobias Schreck, University of Konstanz, Germany, presented research challenges for visual search and analysis in textual and non-textual documents and their evaluation.
The community sessions at CLEF 2012 were organized around the presentation of the activities of other evaluation initiatives and an “Evaluation Clinic” where participants had the possibility of meeting evaluation experts and discussing with them their evaluation problems.
Seven benchmarking activities ran as evaluation labs in CLEF 2012. The results were presented and discussed in Rome in dedicated sessions:
- CHiC (Cultural Heritage in CLEF, new): investigating systematic and large-scale evaluation of cultural heritage digital libraries and information access systems;
- CLEF-IP: studying IR techniques in the patent domain;
- ImageCLEF: proposing experimental evaluation of image classification and retrieval, with a focus on the combination of textual and visual evidence;
- INEX (new in CLEF): evaluating XML retrieval;
- PAN: uncovering plagiarism, authorship and social software misuse;
- QA4MRE: evaluating machine reading systems through question answering and reading comprehension tests;
- RepLab (new): evaluating reputation management technologies.
There was also an exploratory workshop: CLEFeHealth 2012 (new) on cross-language evaluation of methods, applications, and resources for eHealth document analysis.
CLEF 2013 and Beyond
During CLEF 2012 considerable steps were taken to further reshape and improve the overall organization of CLEF. In order to allow for more time for planning and organizing the activities the bidding process for both CLEF 2013 and CLEF 2014 was completed and the host venues of the annual meeting for the next two years were decided.
CLEF 2013 will be hosted by the Technical University of Valencia, Spain, 23-26 September 2013 while CLEF 2014 will be hosted by the University of Sheffield, United Kingdom, 15-19 September 2014.
The Call for Participation in the CLEF2013 Evaluation Labs and the Call for Papers for the CLEF 2013 Conference will both be released in November 2012. Lab registration will open in December 2012 and the expected deadline for the submission of papers to the conference is late April 2013. All details can be found on the CLEF website.
Finally, bids for hosting CLEF 2015 are now open and will close on 5th April 2013. Proposals can be sent to the CLEF Steering Committee Chair at
Links:
CLEF: http://www.clef-campaign.org/
CLEF 2015 Template for Bids: http://www.clef-initiative.eu/documents/71612/87713/CLEF-Initiative-Template_for_bids.docx
PROMISE: http://www.promise-noe.eu/
ELIAS: http://www.elias-network.eu/
Please contact:
Nicola Ferro
University of Padua, Italy
E-mail:
