Toward Rank Correlation as a Measure of Confidence in Information Retrieval Experiment Results

Toward Rank Correlation as a Measure of Confidence in Information Retrieval Experiment Results

Bavdaz, Alenka (TU Delft Electrical Engineering, Mathematics and Computer Science)

Urbano Merino, Julian (mentor)

Delft University of Technology

2018-08-02

In the field of Information Retrieval (IR), test collections are an important part of IR system evaluation. When evaluating IR systems on a test collection, the results may not accurately represent the performance of the systems on topics not contained in that test collection. Therefore, we want to get a sense of the accuracy of results on a given test collection. In this thesis, we use an approach that estimates the accuracy of test collections by estimating rank correlation between the observed and true mean scores of systems. We further evaluate this approach on new data and develop interval estimators as well. This way we provide a better sense of confidence on the system evaluation results by accounting for the inherent variability in sampling topics.

Information Retrieval
Rank Correlation
Reliability
Confidence Intervals

http://resolver.tudelft.nl/uuid:0bc29b72-2e55-4f93-a1d0-fe6885d003dd

Student theses

master thesis

© 2018 Alenka Bavdaz