Egocentric-view fingertip detection for air writing based on convolutional neural networks

Yung Han Chen, Chi Hsuan Huang, Sin Wun Syu, Tien Ying Kuo, Po Chyi Su

Research output: Contribution to journalArticlepeer-review

1 Scopus citations

Abstract

This research investigated real-time fingertip detection in frames captured from the increas-ingly popular wearable device, smart glasses. The egocentric-view fingertip detection and character recognition can be used to create a novel way of inputting texts. We first employed Unity3D to build a synthetic dataset with pointing gestures from the first-person perspective. The obvious benefits of using synthetic data are that they eliminate the need for time-consuming and error-prone manual labeling and they provide a large and high-quality dataset for a wide range of purposes. Following that, a modified Mask Regional Convolutional Neural Network (Mask R-CNN) is proposed, consist-ing of a region-based CNN for finger detection and a three-layer CNN for fingertip location. The process can be completed in 25 ms per frame for 640 × 480 RGB images, with an average error of 8.3 pixels. The speed is high enough to enable real-time “air-writing”, where users are able to write characters in the air to input texts or commands while wearing smart glasses. The characters can be recognized by a ResNet-based CNN from the fingertip trajectories. Experimental results demonstrate the feasibility of this novel methodology.

Original languageEnglish
Article number4382
JournalSensors (Switzerland)
Volume21
Issue number13
DOIs
StatePublished - 1 Jul 2021

Keywords

  • Air-writing
  • Fingertip detection
  • Region-based convolutional neural network
  • Smart glasses

Fingerprint

Dive into the research topics of 'Egocentric-view fingertip detection for air writing based on convolutional neural networks'. Together they form a unique fingerprint.

Cite this