Using Deep Learning Methods in Musical Instrument Fingering for Chinese Puppet Shows

Traditional puppet show is a well-known entertainment program in Taiwan. In the years of 1970’s, witha famous actor and funder, Mr. Hai-Dai Hwang, Traditional puppet show (also known as Bu-Dai-Shi) wasbroadcasted on ordinary TV. The impact of this program is countrywide. Traditional Bu-Dai-Shi show has avery strong influence to many generations of Taiwanese. In line with the development of different TVprograms, Bu-Dai-Shi plays a less important role. However, Bu-Dai-Shi is still very popular in Taiwan.This joint project aims to use computer and sensor technologies, to develop a whole new concept of howto play Bu-Dai-Shi, with a goal to let young people to use VR and AR techniques to play Bu-Dai-Shi. Theproject starts with a 3D modeling technique, which will allow users to capture traditional puppets as 3Dmodels. This step saves time and efforts for users to create new characters. High quality Bu-Dai-Shicharacters will be produced for the use of a series of hand and gesture tracking controls. The second part ofthe joint project uses RGB-D camera and others, to understand the intent of players, especially, with differentapproaches of finger tracking. The result should allow ordinary and professional users to play Bu-Dai-Shiusing bare hand, and to demonstrate show on computers. The third part of the project further uses few typesof sensors, to understand the specific requirements of players. Specific special effects, such as showingweapons, flips of Bu-Dai-Shi, or special movement will be available. The fourth part of the joint project usesfinger-labeling techniques, to develop a series of traditional musical instruments. With bare hands, users canplay traditional instruments to support the performance of large Bu-Dai-Show.