Automatic Detection of Mind-Wandering using Facial Expressions

Automatic Detection of Mind-Wandering using Facial Expressions

Kargul, Radek (TU Delft Electrical Engineering, Mathematics and Computer Science)

Dudzik, B.J.W. (mentor)
Zhang, X. (mentor)
Hung, H.S. (mentor)
Murukannaiah, P.K. (graduation committee)

Delft University of Technology

Computer Science and Engineering

CSE3000 Research Project

2022-06-22

Spending time in front of screens has become an inescapable activity, which might be interrupted by unrelated external causes. While automatic approaches to identify mind-wandering (MW) have already been investigated, past research was done with self-reports or physiological data. This work explores automated detection utilizing solely facial expressions from Mementos data, which comes in the form of webcam recordings, where participants react to music videos. The recordings are annotated with labels indicating perceived MW. Video responses are turned into time series by first extracting facial characteristics, which are encoded with Facial Action Coding System (FACS). Temporal information is represented with 170 temporal features. Classification is conducted with support vector machines (SVM) through a data-level approach and an algorithm-level approach, first by synthesizing data and second by adding class weights to SVM. Both approaches are evaluated with metric scores insensitive to imbalanced data. On average,
results show that detection performs marginally better than by chance. However, the evaluation metric values vary across multiple classification runs, thus the prospect of using the Mementos dataset for automatic MW detection based on only facial expressions is not promising.

mind-wandering
detection
facial expression
machine learning
computer vision

http://resolver.tudelft.nl/uuid:4a66b2e6-8439-41af-a3a2-ea698320e750

Student theses

bachelor thesis

© 2022 Radek Kargul