Experiment design

Given two interactive CLIR systems to be compared, eight searchers and eight topics, the iCLEF experiment design prescribes 1) how to conduct a searching session for a given topic/system/searcher combination, and 2) which topic/system/searcher must be performed, and in what order.

The experiment will use a within-subject design like that used for the (former) TREC interactive track, but with a different number of topics and a different task: each searcher will be presented with all of the 8 topics. The presentation order for topics will be varied systematically to ensure that each topic is searched in a different position, but that the same presentation order is used for each system. The minimal experimental matrix, in the order run, is shown below. For each searcher, the order of the systems and the order of the topics for each system are shown:

Searcher
1
System A / Topic 1
System A / Topic 4
System A / Topic 3
System A / Topic 2
System B / Topic 5
System B / Topic 8
System B / Topic 7
System B / Topic 6
2
System B / Topic 2
System B / Topic 3
System B / Topic 4
System B / Topic 1
System A / Topic 6
System A / Topic 7
System A / Topic 8
System A / Topic 5
3
System B / Topic 1
System B / Topic 4
System B / Topic 3
System B / Topic 2
System A / Topic 5
System A / Topic 8
System A / Topic 7
System A / Topic 6
4
System A / Topic 2
System A / Topic 3
System A / Topic 4
System A / Topic 1
System B / Topic 6
System B / Topic 7
System B / Topic 8
System B / Topic 5
5
System A / Topic 7
System A / Topic 6
System A / Topic 1
System A / Topic 4
System B / Topic 3
System B / Topic 2
System B / Topic 5
System B / Topic 8
6
System B / Topic 8
System B / Topic 5
System B / Topic 2
System B / Topic 3
System A / Topic 4
System A / Topic 1
System A / Topic 6
System A / Topic 7
7
System B / Topic 7
System B / Topic 6
System B / Topic 1
System B / Topic 4
System A / Topic 3
System A / Topic 2
System A / Topic 5
System A / Topic 8
8
System A / Topic 8
System A / Topic 5
System A / Topic 2
System A / Topic 3
System B / Topic 4
System B / Topic 1
System B / Topic 6
System B / Topic 7

Additional searchers can be added in groups of eight using the design above.

Experiment Duration

The experiment will require about three hours for each participant. Each topic will take around 15 minutes: 2 minutes before to examine the topic description, 10 minutes during the search, 3 minutes afterwards to complete the post/search survey. Searchers should not be asked to work for more than an hour without a break. An example schedule for an experimental session would be as follows:

Introductory stuff 10 minutes
Initial survey 5 minutes
Tutorials (2 systems) 30 minutes total
Break 10 minutes
Searching (system A, 4 topics) 60 minutes
Post/system survey 5 minutes
Break 10 minutes
Searching (system B, 4 topics) 60 minutes
Post/system survey 5 minutes
Final survey 10 minutes

 

Fernando López Ostenero - Webmaster
- Javart Web Design and implementation