LightDigit: Embedded Deep Learning Empowered Fingertip Air-Writing with Ambient Light

LightDigit: Embedded Deep Learning Empowered Fingertip Air-Writing with Ambient Light

Liu, Hao (TU Delft Electrical Engineering, Mathematics and Computer Science)

Zuniga, Marco (mentor)
Wang, Q. (mentor)
Yang, J. (mentor)

Delft University of Technology

Electrical Engineering | Embedded Systems

2021-08-21

This thesis introduces LightDigit, a recognition system for fingertip air-writing digits that enables low-cost, privacy-preserving, and in air interaction with pub- lic devices such as touchscreens. The current LightDigit prototype can input digits to public devices with a simple array of photodiodes and with ubiquitous ambient light in our surroundings. The key enabler is an approach detecting and interpreting the dynamic shadow introduced by the movement of the fingertip when a person writes digits in air using a finger. We design two embedded deep learning models, a visible light Transformer (ViLiT) and a customized model ConvRNN with attention pooling, to extract and analyze the spatial and temporal patterns in the dynamic shadow. For training these two models, we build a dataset with 20880 instances of the air-writing for 174 different types of ten digits 0–9. We evaluate LightDigit through extensive experiments in three different scenarios and in two different environments. The evaluation results show that both models are effective, with ConvRNN achieving 98% accuracy in the Within-Subjects scenario. Through model compression, the size of ConvRNN is reduced by 99% with less than a 5% drop in performance. Results further show that LightDigit is robust when ambient light is tilted by 60◦, or when only 33% of photodiodes are working.

http://resolver.tudelft.nl/uuid:afc5471f-c2a3-4ecf-b15e-6edf43a9d22b

2022-09-01

Student theses

master thesis

© 2021 Hao Liu