Dealing with Ties in Rank Correlation

Dealing with Ties in Rank Correlation

Radja, Priyanka (TU Delft Electrical Engineering, Mathematics and Computer Science; TU Delft Software Technology)

Urbano Merino, Julian (mentor)
Hanjalic, Alan (graduation committee)
van Gemert, Jan (graduation committee)

Delft University of Technology

Computer Science | Multimedia Computing

2018-08-27

In the field of Information Retrieval (IR), rankings of systems evaluated under different conditions are often compared to each other. This measure of correspondence between rankings, termed as rank correlation, must accurately capture the scenario for which the correlation is computed. Very often, these rankings may have tied systems, for which new correlation coefficients arise. It is important that these coefficients account for the new scenarios in the presence of ties. It is also important that these coefficients provide some flexibility to the person performing the correlation to introduce artificial ties when items are so close to each other that, for practical purposes, they may be considered as tied. Accounting for these different scenarios of tied items in rankings permits performing per topic comparisons of IR systems, which was earlier limited due to the presence of ties on individual topics. Comparing rankings at the topic level, the expected variability of the rankings can be studied to potentially improve the systems on the topics for which they perform poorer than average. We show the application of these new correlation coefficients with two typical IR experiments.

Information Retrieval
Rank Correlation
Ties

http://resolver.tudelft.nl/uuid:a5707829-748b-405c-b99e-0d0ee0e63a38

Student theses

master thesis

© 2018 Priyanka Radja