Experiment design

Given two interactive Cross-Language QA systems to be compared, eight searchers and sixteen topics, the iCLEF experiment design prescribes 1) how to conduct a searching session for a given topic/system/searcher combination, and 2) which topic/system/searcher combinations must be performed, and in what order.

Topic/system/searcher combinations

The experiment will use a within-subject design like that used for the (former) TREC interactive track, but with a different number of topics and a different task: each searcher will be presented with all of the 16 topics. The presentation order for topics will be varied systematically to ensure that each topic is searched in a different position, but that the same presentation order is used for each system. The minimal experimental matrix, in the order run, is shown below. For each searcher, the order of the systems and the order of the topics for each system are shown:

Questions: 1-16; Systems: A,B; searchers: 1-8 searcher topic order

Searcher 1
A1
A4
A3
A2
A9
A12
A11
A10
B13
B16
B15
B14
B5
B8
B7
B6
Searcher 2
B2
B3
B4
B1
B10
B11
B12
B9
A14
A15
A16
A13
A6
A7
A8
A5
Searcher 3
B1
B4
B3
B2
B9
B12
B11
B10
A13
A16
A15
A14
A5
A8
A7
A6
Searcher 4
A2
A3
A4
A1
A10
A11
A12
A9
B14
B15
B16
B13
B6
B7
B8
B5
Searcher 5
A15
A14
A9
A12
A7
A6
A1
A4
B3
B2
B5
B8
B11
B10
B13
B16
Searcher 6
B16
B13
B10
B11
B8
B5
B2
B3
A4
A1
A6
A7
A12
A9
A14
A15
Searcher 7
B15
B14
B9
B12
B7
B6
B1
B4
A3

A2

A5
A8
A11
A10
A13
A16
Searcher 8
A16
A13
A10
A11
A8
A5
A2
A3
B4
B1
B6
B7
B12
B9
B14
B15


Additional searchers can be added in groups of eight using the design above.

How to conduct a session for a searcher

The experiment will require around three hours for each searcher. In this time, the searcher will
The time to find an answer is 5 minutes per question. Once time expires, the searcher should enter the answer immediately or click an "I do not know" option which will be interpreted as "NIL". Optionally, the user will be asked to report his/her confidence in the answer, choosing between "low" and "high".

See the results dtd and the example submission to check all information that must be stored and sent to iCLEF organizers .

An example schedule for an experimental session would be as follows:

Introductory stuff 10 minutes
Initial survey 5 minutes
Tutorials (2 systems) 30 minutes
Break 10 minutes
Searching (system A, 8 topics) 60 minutes
Post-system survey 5 minutes
Break 10 minutes
Searching (system B, 8 topics) 60 minutes
Post-system survey 5 minutes
Final survey 10 minutes

 

Fernando López Ostenero - Webmaster
- Javart Web Design and implementation