Interactive learning for video content analysis

Interactive learning for video content analysis

Smits, E.A.P.

Hanjalic, A. (mentor)

Electrical Engineering, Mathematics and Computer Science

2008-09-16

We propose a method for identifying segments of a video that represent the events preferred by the user. Possible applications are personalized browsing through music DVDs or smart surveillance systems that can adapt to new circumstances. Requirements for this system are that it is generic and adaptable to the user and to new circumstances. Interactive learning techniques meet these requirements. Interactive learning methods are machine learning methods that involve the user in the learning process in some way. Interactive learning methods do not need information about the problem beforehand and make a system adaptable. For these reasons we propose to apply interactive learning techniques to video content analysis. We chose to implement a user interface that allows users to indicate the relevance of parts of a video by adapting a curve. This curve is used for labelling the underlying audio feature vectors to use as a training set for a classifier. The scenario in which a user adapts a relevance curve while watching a video turned out to be a good way of interacting with the user. The scenario implicates some restrictions for the system, that have turned out to be very strict for the classifier. This classifier needs to be capable of learning efficiently from different sized training sets, handling unbalanced datasets, handling problems with different complexities and not take too much training and classification time. The key findings with respect to our music DVD test scenario are: • Choices about the best classifier and its parameters without any information about events or data can not be optimized because of the lack of prior information. • Support Vector (SV) classifiers need extensive parameter optimization in order to get good results; no optimization means bad results in general and optimizing based on the (small) training set is very time-consuming and can cause overfitting. Because of the constraints in the SV algorithm, this classifier might not handle highly overlapping classes well. These reasons make a SV classifier not a suitable choice for our application. • A Nearest Neighbour (NN) classifier, adapted in such a way that the prior probabilities are taken into account, gave the best overall results on our data. This can be explained by the fact that this classifier has no heavy parameter tuning to perform. • Unbalanced datasets do not have to pose a problem as classifiers can be adapted and post processed to use this information. This same adaptation strategy can be applied for classes that have different classification costs. • User feedback can be used in an 'active learning' manner: Hard to classify samples are labelled and these can be stressed in order to focus more on these samples for better results. To adapt this to surveillance videos the following steps need to be taken: • A general video feature set needs to be chosen and, if necessary, combined with the audio feature set. • Sub sampling needs to be implemented so the system can 'forget' old samples and this way adapt to new circumstances. • The first training is best performed on an offline database or by attaching it to an existing system. This will reduce the training time that is expected to be high because at least a few examples of threat situations need to be labelled, while these situations are rare. Interactive learning is a promising concept to be applied to video concept detection. This thesis analyses the possibilities and restrictions of applying interactive learning to our scenario.

interactive learning
video

http://resolver.tudelft.nl/uuid:51d9ddff-4ff1-494d-8070-c246b073e3d4

TU Delft, Electrical Engineering, Mathematics and Computer Science, Information and Communication Technology (ICT)

Student theses

master thesis

(c) 2008 Smits, E.A.P.