iCLEF is the interactive track of CLEF (Cross-Language Evaluation Forum), an annual evaluation exercise for Multilingual Information Access systems. In iCLEF, Cross-Language search capabilities are studied from a user-inclusive perspective. A central research question is how best to assist users when searching information written in unknown languages, rather than how best an algorithm can find information written in languages different from the query language.
Since 2006, iCLEF has moved from news collections (a standard for text retrieval experiments) in order to explore user behaviour in a collection where the cross-language search necessity arises more naturally for average users. We chose Flickr, a large-scale, web-based image database based on a large social network of WWW users, with the potential for offering both challenging and realistic multilingual search tasks for interactive experiments.
Over the last years, iCLEF participants have typically designed one or more cross-language search interfaces for tasks such as document retrieval, question answering or text-based image retrieval. Interfaces were studied and compared using controlled user populations under laboratory conditions. The main novelty since the iCLEF 2008 track is focusing on the shared analysis of a medium/large search log from a single search interface provided by the iCLEF organizers (i.e. our current focus is on log analysis, rather than on system design). FlickLing has been the search interface provided by iCLEF organizers consisting of a basic cross-language retrieval system for the Flickr image database. We presented it as an online game with a clear goal: the user is given a target image, and she must find it again without any a priori knowledge of the language(s) in which the image is annotated.
The log files generated during the last two campaigns and other associated data are freely available.
help file. Please, read it carefully.
help file. Please, read it carefully.