TY - GEN
T1 - Measuring the variability in effectiveness of a retrieval system
AU - Hosseini, Mehdi
AU - Cox, Ingemar J.
AU - Millic-Frayling, Natasa
AU - Vinay, Vishwa
PY - 2010
Y1 - 2010
N2 - A typical evaluation of a retrieval system involves computing an effectiveness metric, e.g. average precision, for each topic of a test collection and then using the average of the metric, e.g. mean average precision, to express the overall effectiveness. However, averages do not capture all the important aspects of effectiveness and, used alone, may not be an informative measure of systems' effectiveness. Indeed, in addition to the average, we need to consider the variation of effectiveness across topics. We refer to this variation as the variability in effectiveness. In this paper we explore how the variance of a metric can be used as a measure of variability. We define a variability metric, and illustrate how the metric can be used in practice.
AB - A typical evaluation of a retrieval system involves computing an effectiveness metric, e.g. average precision, for each topic of a test collection and then using the average of the metric, e.g. mean average precision, to express the overall effectiveness. However, averages do not capture all the important aspects of effectiveness and, used alone, may not be an informative measure of systems' effectiveness. Indeed, in addition to the average, we need to consider the variation of effectiveness across topics. We refer to this variation as the variability in effectiveness. In this paper we explore how the variance of a metric can be used as a measure of variability. We define a variability metric, and illustrate how the metric can be used in practice.
UR - https://www.scopus.com/pages/publications/77954435144
U2 - 10.1007/978-3-642-13084-7_7
DO - 10.1007/978-3-642-13084-7_7
M3 - Conference contribution
AN - SCOPUS:77954435144
SN - 3642130836
SN - 9783642130830
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 70
EP - 83
BT - Advances in Multidisciplinary Retrieval - First Information Retrieval Facility Conference, IRFC 2010, Proceedings
T2 - 1st Information Retrieval Facility Conference, IRFC 2010
Y2 - 31 May 2010 through 31 May 2010
ER -