Project Details


Sign Language Translation is considered to be one of the important applicationsof Human-Computer-Interaction, Natural Language Processing, and VirtualReality Animation. This is a two-year continuous proposal, based on our twoprojects sponsored by the MOST (109-2221-E-008 -067 and 110-2221-E-008 -082). Our first 3-year proposal was only approved for one year, followed byanother accepted proposal for the second year. In this new two-year proposal, wewill enhance our model to be used for practical Taiwan Sign Languagetranslation. As shown in the project report of the first year, and the related worksection in this proposal, our WER (Word Error Rate) is superior to most researchoutcome in the literature. However, only public datasets were used for testing andcomparison. Due to budget limitation, we are not able to hire Taiwan SignLanguage professionals for data collection.In this new two-year proposal, we will complete the following issues:•Data collection of Taiwan Sign Language•Revise current model to reduce parameters•Demo website for direct translation from Taiwan Sign Language to TraditionalChinese Text•Translation of text to skeleton sign language•Mapping skeleton sign language to avatar motion•Demo website of bi-directional translation under metaverseIf the project is approved with a reasonable support, we will hire professionals fordata collection, and to construct a public available website for Taiwan SignLanguage Translation. Our current deep learning model takes more than twoweeks of training time using one 3090 GPU. It is very time consuming fordeveloping a good and practical translation system. We are trying to reduceparameters for an efficient and effective system. However, the current testingmodel is able to translate sign language in a few seconds. We have developed atesting website. However, due to copyright issue, we are not able to open thewebsite now (i.e., we are using Chinese Signa Language for demo). We areworking on translation from Chinese text to avatar presentation of sign language.Currently, we use publicly available avatar. However, we are developing our ownavatars using VR techniques. We hope that the completed system can contributeto Taiwan Sign Language users, with a hope that the translation technique canbe used with different sign languages.
Effective start/end date1/08/2231/07/23

UN Sustainable Development Goals

In 2015, UN member states agreed to 17 global Sustainable Development Goals (SDGs) to end poverty, protect the planet and ensure prosperity for all. This project contributes towards the following SDG(s):

  • SDG 4 - Quality Education
  • SDG 11 - Sustainable Cities and Communities
  • SDG 12 - Responsible Consumption and Production
  • SDG 17 - Partnerships for the Goals


  • Taiwan Sign Language Translation
  • Automatic Language Translation
  • Sign Language Training
  • Text to Sign Language Translation
  • Deep Learning


Explore the research topics touched on by this project. These labels are generated based on the underlying awards/grants. Together they form a unique fingerprint.