iCLEF 2009

Important note

Please notice flickling databases has been reset on May 12 2009. If you already participated before that date, you need to re-register again. Sorry for the inconvenience!

Guidelines

1) Search Task definition

The task is known-item image retrieval from Flickr. The user is given an image (not annotated) and the goal is to find the image again from Flickr using flickling, the search engine provided by iCLEF organizers. The user does not know in advance in which languages the image is annotated. Therefore, searching in multiple languages is essential to get optimal results.

The task is organized as an online game: the more images you find, the higher you will be ranked. In case of ties, the ranking will also depend on precision (number of images found / number of images attempted) and search time spent in finding the images (less time is better). There will be a "Hall of Fame" with a rank of the best users. At any time whilst searching, the user will be allowed to quit the search (skip to next image) or ask for hints (a keyword or an indication on the target language).

2) Search interface

The iCLEF organizers provide a basic cross-language search front-end to Flickr, with the following basic functionalities:

• Multilingual search, query in one language, search results in up to six languages: Dutch, English, French, German, Italian and Spanish.
• Cross-language search is performed via term-to-term translations between six languages using free dictionaries.
• The term-to-term translation facility selects the best target translations according to: i) Flickr related terms, and ii) string similarity between the source and target words.
• Allows the user pick/remove translations, and add their own translations (which are added into the user's personal dictionary).
• Provides search suggestions and relevance feedback facilities.
• Controls the game-like features of the task: user registration and user profiles, flow of images, recording of session logs, hall of fame, etc.

flickling is up and running but first check the tutorial to learn its functionalities.

For any questions or support using the interface, please contact the .

3) Search logs

Search logs will be harvested from the search interface described above. Our intention is to have two main user groups:

1. CLEF participants, who will be asked to interact with the system (the best team will receive an award at CLEF 2009 workshop). Besides contributing to the search logs, we believe that this experience will be an interesting hands-on experience for researchers working on CLIR that have never actually been users of a CLIR system.
2. Flickr/web users at large. The game will be publicized in Flickr forums in order to increase the size of the search log.

4) What are iCLEF participants supposed to do?

iCLEF participants can essentially do two things:

• Search log analysis: participants will have access to the search logs (logs generated during iCLEF 2008 will also be provided), and can freely perform data mining studies on them. Initial examples are: looking for differences in search behaviour according to language skills, or looking for correlations between search success and search strategies, etc.
• Interactive experiments: participants can recruit their own users and conduct their own experiments with the interface. For instance, they can recruit a set of users with passive abilities and another with active abilities in certain languages and, besides studying the search logs, they can perform observational studies on how they search, conduct interviews, etc.

iCLEF organizers will provide assistance to define the appropriate user groups, image lists, etc. within the common search interface. Besides these two options, and given the open spirit of iCLEF, we are open to groups having their own plans (for instance, their own interface designs) as long as they do not change the shared task (known-item search on Flickr).

5) OK, I'm interested. How do I join?

Please e-mail the stating your interest in the task, and go to the main CLEF page and follow the general registration procedure.

6) Schedule

Registration is now open. The only hard deadline is August 30, when the paper for the CLEF workshop notes is due. Meanwhile, iCLEF organizers will contact participants to assist them, promote discussions on the experimental design and retrieve information on the experiments for the iCLEF overview paper.

• Registration: Open now
• Search interface: flickling is available now!
• Search logs released: June 30
• Paper for working notes due: August 30
• CLEF Workshop: September 30 to October 2 in Corfu, Greece

7) More Information

You can get data, examples and logs generated during the 2008 and 2009 campaigns at this iCLEF repository.

And feel free to for any question regarding iCLEF.

Home Guidelines Task Definition Search Interface Search Logs What are participants supposed to do? How to participate Schedule More Information Past events Coordinators CLEF