iCLEF 2008

Guidelines

1) Search Task definition

The task is known-item image retrieval from Flickr. The user is given an image (not annotated) and the goal is to find the image again from Flickr using the system provided by iCLEF organizers. The user does not know in advance in which languages the image is annotated; therefore searching in multiple languages is essential to get optimal results.

The task is organized as an online game: the more images you find, the higher you will be ranked. In case of ties, the ranking will also depend on precision (number of images found / number of images attempted) and search time (less time is better). There will be a "Hall of Fame" with a rank of the best users. At any time whilst searching, the user will be allowed to quit the search (skip to next image) or ask for hints (a keyword or an indication on the target language).

2) Search interface

The iCLEF organizers provide a basic cross-language search front-end to Flickr, with the following basic functionalities:

• Multilingual search, query in one language, search results in up to six languages.
• Cross-language search is performed via term-to-term translations between six languages using free dictionaries.
• The term to term translation facility selects the best target translations according to (i) Flickr related terms, (ii) string similarity between the source and target words.
• Allows the user pick/remove translations, and add their own translations (which go into a personal dictionary).
• Provides search suggestions and relevance feedback facilities.
• Controls the game-like features of the task: user registration and user profiles, flow of images, recording of session logs, hall of fame, etc.

NEW!! Flickling Interface available

For any questions or support using the interface, write to and you will normally receive a reply in around 24 hours.

3) Search logs

Search logs will be harvested from the search interface described above. Our intention is to have two main user groups:

1. CLEF participants, who will be asked to interact with the system (the best team will receive an award at CLEF 2008). Besides contributing to the search logs, we believe that this experience will be an interesting hands-on experience for researchers working on CLIR that have never actually been users of a CLIR system.
2. Flickr/web users at large. The game will be publicized in Flickr forums in order to increase the size of the search log.

4) What are iCLEF participants supposed to do?

iCLEF participants can essentially do two things:

• Search log analysis: participants will have access to the search logs, and can freely perform data mining studies on them. Initial examples are: looking for differences in search behaviour according to language skills, or looking for correlations between search success and search strategies, etc.
• Interactive experiments: participants can recruit their own users and conduct their own experiments with the interface. For instance, they can recruit a set of users with passive abilities and another with active abilities in certain languages and, besides studying the search logs, they can perform observational studies on how they search, conduct interviews, etc.

iCLEF organizers will provide assistance to define the appropriate user groups, image lists, etc. within the common search interface. Besides these two options, and given the open spirit of iCLEF, we are open to groups having their own plans (for instance, their own interface designs) as long as they do not change the shared task(known-item search on Flickr).

5) OK, I'm interested. How do I join?

Please e-mail the stating your interest in the task, and go to the main CLEF page and follow the general registration procedure.

6) Schedule

Registration is now open. The only hard deadline is August 15, when the paper for the CLEF workshop notes is due. Meanwhile, iCLEF organizers will contact participants to assist them, promote discussions on the experimental design and retrieve information on the experiments for the iCLEF overview paper.

• Registration: Open now
• Search interface: available now
• Search logs released: June 10
• Paper for working notes due: August 15
• CLEF Workshop: September 18-19, Aarhus

7) More Information

Read this workshop paper for more details about the task. Do not hesitate to for any question regarding iCLEF.

Home Guidelines Task Definition Search Interface Search Logs What are participants supposed to do? How to participate Schedule More Information Past events Coordinators CLEF