Using semantic relatedness to improve the evaluation of multi-label classifiers

Using semantic relatedness to improve the evaluation of multi-label classifiers

Deloo, C.P.P.

Houben, G.J.P.M. (mentor)

Electrical Engineering, Mathematics and Computer Science

Web Information Systems

Computer Science, Software Technolgy

2013-03-19

Different evaluation techniques can be used in order to assess the quality of a multi-label classification task. The most commonly used performance measures in classification are the classic information retrieval notions of precision and recall. These measures assume independence between the target labels meaning that the assignment of a label to an item can either be evaluated as correct or incorrect and not somewhere in between. The drawback of these binary evaluation measures is that they do not take into account the difference in semantics between the concepts depicted by the labels. For example, an item classified as plant instead of flower would be considered as wrong although the concepts are semantically related and a flower is a specialisation of a plant. A way of improving the evaluation of a classification would be to also consider the degree of semantic relatedness between concepts. The work in this thesis focuses on improving the evaluation of a real-world multi-label classification task by using the notion of semantic relatedness. The labels of this classification task represent concepts originating from a hierarchically structured thesaurus. Intuitively, the concepts that are close to each other in the graph are more related than the ones that are more distant from each other. Several techniques that measure semantic relatedness based on this principle have been proposed in scientific literature. This work investigates the performance of such techniques in the evaluation of multi-label classifications by assessing them against human judgements of relatedness. It is shown that an evaluation incorporating semantic relatedness is likely to be more accurate than the traditional binary evaluations. The knowledge on classifier assessment acquired in this study is applied in practice in the form of an evaluation framework comprising several techniques for measuring and analysing the performance of classifiers.

classifier evaluation
semantic relatedness
multi-label classification

http://resolver.tudelft.nl/uuid:8fb5b6f6-93a6-4aed-a446-013125541ff9

2013-03-19

Student theses

master thesis

(c) 2013 Deloo, C.P.P.