Since 2000 the Cross-Language Evaluation Forum (CLEF) has played a successful role in stimulating research and promoting evaluation in a wide range of key areas in the information retrieval domain, becoming well-known in the international IR community. The results had traditionally been presented and discussed at annual workshops in conjunction with the European Conference for Digital Libraries.
CLEF 2010 represented a radical innovation of this traditional format. The aim is to continue with the benchmarking evaluation experiments, but also to provide more space for analysis, reflection and discussion. It was thus organised as an four-day event: a peer-reviewed conference followed by a series of laboratories and workshops.
Scenes from CLEF 2010
CLEF 2010: The Conference
The conference aimed at advancing research into the evaluation of complex multimodal and multilingual information systems. Topics covered included experimental collections and datasets, evaluation methodologies, resources and tools. Two keynote talks by Norbert Fuhr, University of Duisburg-Essen and Ricardo Baeza-Yates, Yahoo! Research and Universitat Pompeu Fabra discussed future directions for experimental evaluation from academic and industrial perspectives. Reports from other major evaluation initiatives were presented: the Text REtrieval Conference, USA, the NII-NACSIS Test Collection for IR Systems, Japan, the INitiative for the Evaluation of XML Retrieval, Australia, the Information Retrieval Evaluation Seminar, Russia, and the Forum for Information Retrieval Evaluation in India. There were also two panels. The first presented the objectives of a new EU FP7 Network of Excellence, PROMISE, which aims at advancing the experimental evaluation of complex multimedia and multilingual information systems. In the second, researchers responsible for creating and running some of the main evaluation initiatives of the last two decades, discussed what has been achieved so far by these initiatives and, more importantly, what still needs to be done.
CLEF 2010: The Labs and The Workshops
The laboratories continue and expand on the traditional CLEF track philosophy. Two different forms of labs are offered: benchmarking activities proposing evaluation tasks and comparative performance analyses, and workshop-style labs that explore issues of information access evaluation and related fields.
There were five benchmarking activities:
- CLEF-IP ‘10, sponsored by the Information Retrieval Facility (IRF) in Vienna investigated patent retrieval in a multilingual context
- ImageCLEF 2010 offered several tasks for both context and content-based retrieval of medical images from the journal of Radiology and Radiographics and photographic images from Flickr and Wikipedia
- PAN, sponsored by Yahoo Research, investigated the detection of plagiarism and of Wikipedia vandalism
- QA@CLEF 2010 used the Corpus of the European parliament and offered monolingual question answering tasks for English, French, German, Italian, Portuguese, Spanish and Romanian
- WePS (Web People Search) focused on person name ambiguity and person attribute extraction and online reputation management
And two exploratory workshops:
- CriES addressed the problem of multi-lingual expert search in social media environments.
- LogCLEF studied search behaviour in a multilingual context via query analysis and classification.
CLEF 2010 was hosted by the Department of Information Engineering, University of Padua, Italy and was partially supported by the PROMISE project. Approximately 140 researchers participated in the event; with most of them staying for the full four days
All the presentations given at the conference and in the workshops can be found on the CLEF 2010 website
CLEF 2011
CLEF 2011 will be hosted by the University of Amsterdam in September 2011. A Call for Lab proposals will be issued at the beginning of October 2010 and an initial Call for Papers will be released in December 2010.
Links:
CLEF 2010: http://www.clef2010.org/
PROMISE: http://www.promise-noe.eu/
Please contact:
Nicola Ferro
University of Padua, Italy
E-mail:
