Towards Real-Time Object Removal and Inpainting Through A Diminished Reality Application For Smartphones

Towards Real-Time Object Removal and Inpainting Through A Diminished Reality Application For Smartphones

Cording, Henry (TU Delft Electrical Engineering, Mathematics and Computer Science)

Usta, B. (mentor)
Weinmann, M. (mentor)
Eisemann, E. (mentor)
Erkin, Z. (graduation committee)

Delft University of Technology

Computer Science and Engineering

CSE3000 Research Project

2022-06-27

Diminished reality (DR) is an extension of augmented reality (AR) in which real objects are concealed, removed, or replaced. State of the art DR implementations are written and evaluated on desktop platforms, and those which are aimed at smartphones use auxiliary data or hardware for image completion. This paper introduces a DR app for Android smartphones which can select, track, and remove objects in an unrestricted environment from live video at interactive speed. The application relies exclusively on RGB frames from the phone's camera as input to the DR pipeline. Multiple algorithms for object selection and tracking, as well as a patch-based inpainting approach were adapted. Selection and tracking of objects was very reliable at interactive speed. Inpainting led to convincing results for planar surfaces and unstructured textures, often delivering acceptable results even in more complicated scenes. Future optimizations are required for real-time performance of DR algorithms even on a flagship smartphone, as the hardware was a constraining factor. The performance of inpainting in particular needs to be improved to achieve acceptable frame rates.

diminished reality
augmented reality
mobile
real-time
interactive

http://resolver.tudelft.nl/uuid:e439dc4d-46c8-4d4d-8853-9746f93ed697

Student theses

bachelor thesis

© 2022 Henry Cording