by Nicola Ferro and Paolo Rosso
The CLEF Initiative is structured in two main parts: a series of Evaluation Labs, to conduct evaluation of information access systems and a peer-reviewed Conference on a broad range of issues on evaluation. The annual CLEF events have been partially supported by the EU FP7 PROMISE project (contract n. 258191) and by the ELIAS RNP network.
{jcomments on}CLEF 2013: Information Access Evaluation meets Multilinguality, Multimodality, and Visualization
The annual meeting of the CLEF Initiative was hosted this year by the Universitat Politècnica de València, Spain, 23-26 September as a 4 days event, attended by more than 200 participants, in which conference presentations, laboratory meetings, workshops, and community sessions were smoothly interleaved to provide a continuous stream of discussions on the different facets of experimental evaluation.
22 papers were accepted for the Conference and published by Springer in their Lectures Notes for Computer Science (LNCS) series. Two keynote speakers highlighted important developments in the field of evaluation. Evangelos Kanoulas, Google Zurich, Switzerland, focused on how to model variance in topics, collections, system and measures when evaluating systems. Rada Mihalcea, University of Michigan, USA, presented several scenarios where multilinguality can be pivoted to improve natural language processing performances instead of being a barrier to be crossed.
Nine benchmarking activities ran as evaluation labs in CLEF 2013. The results were presented and discussed in Valencia in dedicated sessions:
- CHiC (Cultural Heritage in CLEF): investigating systematic and large-scale evaluation of cultural heritage digital libraries and information access systems;
- CLEFeHealth (new): aiming at developing processing methods and resources to enrich difficult-to-understand health text as well as their evaluation setting;
- CLEF-IP: studying IR techniques in the patent domain;
- ImageCLEF: proposing experimental evaluation of image classification and retrieval, with a focus on the combination of textual and visual evidence;
- INEX: evaluating XML retrieval;
- PAN: uncovering plagiarism, authorship and social software misuse;
- QA4MRE: evaluating machine reading systems through question answering and reading comprehension tests;
- QALD-3 (Question Answering over Linked Data, new): a benchmarking activity on question answering over linked data;
- RepLab: evaluating reputation management technologies.
There was also an exploratory workshop CLEF-ER: a workshop on multilingual annotation of named entities and terminology resources acquisition.
CLEF 2014 and Beyond
During CLEF 2013 considerable steps were taken to further reshape and improve the overall organization of CLEF. In particular, it has been decided to establish a legal entity to support the organization of the CLEF initiative.
CLEF 2014 will be hosted by the University of Sheffield, UK, 15-19 September 2014 while CLEF 2015 will be hosted by the Institut de Recherche en Informatique de Toulouse (IRIT), France, in early September 2015.
Finally, bids for hosting CLEF 2016 are now open and will close on 7th April 2014. Proposals can be sent to the CLEF Steering Committee Chair at
Links:
CLEF: http://www.clef-initiative.eu/
CLEF 2013 Web site: http://www.clef2013.org/
CLEF 2014 Web site: http://clef2014.clef-initiative.eu/
PROMISE: http://www.promise-noe.eu/
ELIAS: http://www.elias-network.eu/
Please contact:
Nicola Ferro
CLEF Steering Committee Chair
University of Padua, Italy
E-mail:
