Design of a real-time computer-vision and gaze-based pedestrian warning system

Design of a real-time computer-vision and gaze-based pedestrian warning system

Thirunavukkarasu Kumaaravelu, Lokkeshver (TU Delft Mechanical, Maritime and Materials Engineering)

de Winter, J.C.F. (mentor)
Dodou, D. (mentor)
Onkhar, V. (mentor)
Bazilinskyy, P. (mentor)
Tabone, W. (mentor)
Eisma, Y.B. (graduation committee)

Delft University of Technology

Mechanical Engineering

2022-05-31

Vulnerable road users account for more than 50% of traffic fatalities, and among these, pedestrians are the most susceptible to fatalities due to their distraction and misperception of other road users. To mitigate their plight, systems that warn drivers and pedestrians in case of a possible collision have been developed. Among systems that focus on the pedestrian's perspective, existing concepts are capable of predicting collisions but lack elements that monitor the visual attention of pedestrians. We address this gap by developing a gaze-based pedestrian warning system based on the Tobii Pro Glasses 2, a head-mounted eye-tracker. The system consists of: (1) a custom trained fast neural network (YOLO v4) on the KITTI object detection dataset that processes the video feed of the eye-tracker to detect approaching vehicles and (2) a module that uses the pedestrian's gaze to identify whether their attention falls on the closest moving vehicle that is approaching the pedestrian, both in real-time. If the pedestrian does not look at the approaching vehicle, they are given an auditory alert that warns them of a possible collision. In a pilot study conducted on a busy road in an urban environment, the system was evaluated under different pedestrian walking speeds and gaze behaviours to test the algorithm's robustness. The pilot study revealed that our system alerted the inattentive pedestrian with an accuracy of 67%. The mean vehicle detection accuracy and a mean moving vehicle identification accuracy from the pilot were 93% and 60%, respectively, a promising result given the use of only a mono camera. Despite the use of computer vision techniques, the system worked at an inference speed of 50 FPS due to the multi-processing capabilities of our algorithm. Our efforts are a first step in developing pedestrian warning systems based on eye-tracking technology to improve road safety in the future. The algorithm (Python-based) code used for this work has been made publicly available.

Pedestrian Safety
Object detection
Eye tracking
real-time systems
warning system

http://resolver.tudelft.nl/uuid:d52f49ce-1edb-4dc1-94f7-a5617e79ee42

Student theses

master thesis

© 2022 Lokkeshver Thirunavukkarasu Kumaaravelu