A TinyML system for gesture detection using 3D pre-processed data

A TinyML system for gesture detection using 3D pre-processed data

van den Broek, Sem (TU Delft Electrical Engineering, Mathematics and Computer Science)

Wang, Q. (mentor)
Yang, M. (mentor)
Zhu, R. (mentor)
Venkatesha Prasad, Ranga Rao (graduation committee)

Delft University of Technology

Computer Science and Engineering

CSE3000 Research Project

2023-07-03

Visible light sensing is a field of research that creates new possibilities for human-computer interaction. This research shows the viability of designing a system for detecting hand gestures using a cost-effective detection circuit employing 3 light-sensitive photodiodes. The way this research shows viability is by developing a machine-learning model that works on 3D-structured sensor data that is able to distinguish 10 different gestures and deploying the model on a standalone Arduino Nano 33 BLE microcontroller controlling the system. Using a combination of Convolutional Neural Networks and Recurrent Neural Networks it is possible to deploy a model called ConvLSTM-128 that achieves an accuracy of 70% on a dataset of limited size. This research acknowledges that the achieved accuracy is not suitable for real-world use, but concludes by outlining steps that could help future research in increasing the accuracy. Furthermore, an analysis of the 10 gestures shows that in order to improve accuracy, the way some gestures are performed might need alteration. Finally, a model size of around 140Kb and an inference time of 660ms show that this model is compact and fast enough to be deployed in real-world applications.

TinyML
LSTM
ConvLSTM
Arduino Nano 33 BLE
Gesture Recognition
Machine Learning (ML)

http://resolver.tudelft.nl/uuid:a0358845-34c0-4169-94a1-680e1bb18dff

Student theses

bachelor thesis

© 2023 Sem van den Broek