Multi-Modal Deep Learning-Based Violin Bowing Action Recognition

Bao Yun Liu, Yi Hsin Jen, Shih Wei Sun, Li Su, Pao Chi Chang

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

In this paper, a deep learning-based violin action recognition is proposed. By fusing the sensing signals from depth camera modality and inertial sensor modalities, violin bowing actions can be recognized by the proposed deep learning scheme. The actions performed by a violinist are captured by a depth camera, and recorded by wearable sensors on the forearm of a violinist. In the proposed system, 3D convolution neural network (3D-CNN) and long short-term memory (LSTM) deep learning algorithms are adopted to generate the action models from depth camera modality and inertial sensor modalities. The features and models obtained from multi-modalities are used to classify different violin bowing actions. A fusion process from different modalities can achieve satisfactory recognition accuracy. In this paper, we generate a violin bowing actions dataset for the preliminary study and the system performance evaluation.

Original languageEnglish
Title of host publication2020 IEEE International Conference on Consumer Electronics - Taiwan, ICCE-Taiwan 2020
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781728173993
DOIs
StatePublished - 28 Sep 2020
Event7th IEEE International Conference on Consumer Electronics - Taiwan, ICCE-Taiwan 2020 - Taoyuan, Taiwan
Duration: 28 Sep 202030 Sep 2020

Publication series

Name2020 IEEE International Conference on Consumer Electronics - Taiwan, ICCE-Taiwan 2020

Conference

Conference7th IEEE International Conference on Consumer Electronics - Taiwan, ICCE-Taiwan 2020
Country/TerritoryTaiwan
CityTaoyuan
Period28/09/2030/09/20

Fingerprint

Dive into the research topics of 'Multi-Modal Deep Learning-Based Violin Bowing Action Recognition'. Together they form a unique fingerprint.

Cite this