Track: The iCLEF challenge

The goal of  iCLEF is to study the interactive aspects of Cross-Language Information Retrieval systems. Standard CLEF, NTCIR and Cross-Language TREC tasks evaluate the ability of systems to automatically retrieve target-language(s) documents from source-language queries; iCLEF evaluates how well systems help users locate and identify relevant foreign-language documents.

Essentially, interactive CLIR systems can help users to:

- Formulate and/or translate the query.
- Refine the formulation and/or translation of the query depending on the outcome of the system.
- identify foreign-language documents as relevant (cross-language document selection)

Research teams participating in iCLEF are supposed to study some of these issues by comparing two systems in a search task involving a number of topics (provided by iCLEF) and a number of searchers (recruited locally by the participant team). The two systems should differ in the facilities provided for any of the tasks listed above. The iCLEF experiment design will allow groups to estimate the effect of system differences by suppressing the (additive) effects of participant and topic, and by reducing somewhat the effects of interactions between these factors.

There are two possibilities for the search task that the compared systems support:

- If the focus is on query formulation, translation and refinement, the search task consists in finding as many relevant documents as possible for every topic, system and searcher combination prescribed by the iCLEF experiment design .

- If the focus is on cross-language document selection, the (partial) search task may consist on scanning a fixed ranked list of foreign-language documents (returned by some CLIR system), and selecting the relevant ones. Again, this task will be performed for every topic/system/searcher combination prescribed in the shared experiment design .

Participating teams should focus on one of the following user groups (if both groups are studied, separate experiments should be run for each):
  1. searchers with passive language abilities in the foreign language (i.e. that can at least roughly understand documents in that language, but cannot form accurate queries in that language without assistance). For example, a native speaker of Italian that is searching Spanish documents might be a member of this user group.
  2. searchers with no useful language abilities in the foreign language. For example, a monolingual Spanish speaker that is searching German documents might be a member of this user group.

In 2003, iCLEF has eight broad questions, each of which asks about a topic that includes multiple aspects. The questions are selected from the CLEF-2002 topics that hit a reasonable number of documents in all CLEF languages.

Track - How to - Data provided - Data to be submitted - Evaluation - Schedule

How to participate

Research groups interested in joining the iCLEF 2003 task should follow these steps:

1. Register as CLEF participants (follow instructions in Upon registration, every participant will receive instructions to download the appropriate document collections from the CLEF ftp site.

2. E-mail the track organizers (Doug Oard and Julio Gonzalo) indicating their wish to participate and the languages (user and document languages) that will be used in the intended study. Once registration for CLEF is confirmed, participants will receive instructions to download iCLEF-specific data.

No formal coordination of hypotheses or comparison of systems across sites is planned for iCLEF 2003, but groups are encouraged to seek out and exploit synergies. As a first step, groups are strongly encouraged to make the focus of their planned investigations known to other track participants as soon as possible, preferably via the track listserv at Contact Julio Gonzalo ( to join the list.

The CLEF 2002 working notes contain descriptions of experiments carried out in iCLEF 2002 that may be used as a guidance for iCLEF 2003 experiments.

Track - How to - Data provided - Data to be submitted - Evaluation - Schedule

Data Provided by the Organizers

Track - How to - Data provided - Data to be submitted - Evaluation - Schedule

Data to be Submitted to the Organizers

For every search (topic/searcher/system combination), two types of data will be collected:
  1. [REQUIRED] The list of documents that are found to be relevant by the user, for every topic/system/user combination attempted.
  2. [OPTIONAL] If query translation is supported, the top 100 documents in every ranked list generated by the system during interactive query translation (only one list is required if the user does not reformulate the query and search again).

Information about where to submit this data will be provided to the participating teams about one month before the submission deadline. The format of data to be submitted can be downloaded here .

Track - How to - Data provided - Data to be submitted - Evaluation - Schedule

Evaluation Measures

The CLEF-2002 relevance assessment for the chosen document language will be used as ground truth:

F_ALPHA = 1/[(ALPHA/P + (1-ALPHA)/R]

where P is precision and R is recall, measured over the set of documents retrieved and found to be relevant by the user, given an interactive CLIR system and a search topic.
Values of ALPHA below 0.5 emphasize recall, values greater than 0.5 emphasize precision. ALPHA=0.8 will be the default value, modeling the case in which missing some relevant documents would be less objectionable than paying to obtain fluent translations of many documents that later turn out not to be relevant.

The nature of any further detailed analysis is up to each site, but sites are strongly encouraged to take advantage of the experimental design and undertake exploratory data analysis to examine the patterns of correlation and interaction among factors that can be observed in the experiment, and to explore the potential for gaining additional insight through alternative evaluation measures. The computation of analysis of variance (ANOVA), where appropriate, can provide useful insights into the separate contributions of searcher, topic and system as a first step in understanding why the results of one search are different from those of another.

Track - How to - Data provided - Data to be submitted - Evaluation - Schedule


30 March Topics and documents available
15 May Deadline for submission of data
7 July
Results submitted to participants
20 July
Submit notebook papers
21-22 August CLEF Workshop in Trondheim


Fernando López Ostenero - Webmaster
- Javart Web Design and implementation