The experiment will use a within-subject design like that used for the (former) TREC interactive track, but with a different number of topics and a different task: each searcher will be presented with all of the 16 topics. The presentation order for topics will be varied systematically to ensure that each topic is searched in a different position, but that the same presentation order is used for each system. The minimal experimental matrix, in the order run, is shown below. For each searcher, the order of the systems and the order of the topics for each system are shown:

Questions: 1-16; Systems: A,B; searchers: 1-8 searcher topic order

Searcher 1 |
A1 |
A4 |
A3 |
A2 |
A9 |
A12 |
A11 |
A10 |
B13 |
B16 |
B15 |
B14 |
B5 |
B8 |
B7 |
B6 |

Searcher 2 |
B2 |
B3 |
B4 |
B1 |
B10 |
B11 |
B12 |
B9 |
A14 |
A15 |
A16 |
A13 |
A6 |
A7 |
A8 |
A5 |

Searcher 3 |
B1 |
B4 |
B3 |
B2 |
B9 |
B12 |
B11 |
B10 |
A13 |
A16 |
A15 |
A14 |
A5 |
A8 |
A7 |
A6 |

Searcher 4 |
A2 |
A3 |
A4 |
A1 |
A10 |
A11 |
A12 |
A9 |
B14 |
B15 |
B16 |
B13 |
B6 |
B7 |
B8 |
B5 |

Searcher 5 |
A15 |
A14 |
A9 |
A12 |
A7 |
A6 |
A1 |
A4 |
B3 |
B2 |
B5 |
B8 |
B11 |
B10 |
B13 |
B16 |

Searcher 6 |
B16 |
B13 |
B10 |
B11 |
B8 |
B5 |
B2 |
B3 |
A4 |
A1 |
A6 |
A7 |
A12 |
A9 |
A14 |
A15 |

Searcher 7 |
B15 |
B14 |
B9 |
B12 |
B7 |
B6 |
B1 |
B4 |
A3 |
A2 |
A5 |
A8 |
A11 |
A10 |
A13 |
A16 |

Searcher 8 |
A16 |
A13 |
A10 |
A11 |
A8 |
A5 |
A2 |
A3 |
B4 |
B1 |
B6 |
B7 |
B12 |
B9 |
B14 |
B15 |

Additional searchers can be added in groups of eight using the design above.

- Receive general instructions about the task.
- Fulfill an number of pre and post search questionnaires.
- Receive tutorials and practice with both systems.
- Search 8 questions with system A and another 8 questions with system B.

See the results dtd and the example submission to check all information that must be stored and sent to iCLEF organizers .

An example schedule for an experimental session would be as follows:

Introductory stuff | 10 minutes |

Initial survey | 5 minutes |

Tutorials (2 systems) | 30 minutes |

Break | 10 minutes |

Searching (system A, 8 topics) | 60 minutes |

Post-system survey | 5 minutes |

Break | 10 minutes |

Searching (system B, 8 topics) | 60 minutes |

Post-system survey | 5 minutes |

Final survey | 10 minutes |

Fernando López Ostenero |