Evaluation of Query Performance Prediction Methods by Range.
Joaquín Pérez-Iglesias, Lourdes Araujo.
Proc. Int. Symposium on String Processing and Information Retrieval (SPIRE 10)
LNCS 6393, pp. 225-236, Springer (2010)
During the last years a great number of Query Performance Prediction methods have been proposed.
However, this explosion of prediction method proposals have not been paralleled by an in-depth
study of suitable methods to evaluate these estimations. In this paper we analyse the current
approaches to evaluate Query Performance Prediction methods, highlighting some limitations they
present. We also propose a novel method for evaluating predictors focused on revealing the different
performance they have for queries of distinct degree of difficulty. This goal can be achieved by
transforming the prediction performance evaluation problem into a classification task, assuming
that each topic belongs to a unique type based on their retrieval performance. We compare the
different evaluation approaches showing that the proposed evaluation exhibits a more accurate
performance, making explicit the differences between predictors for different types of queries.