Exploiting unlabeled internal data in conditional random fields to reduce word segmentation errors for Chinese texts

Richard Tzong Han Tsai, Hsi Chuan Hung, Hong Jie Dai, Wen Lian Hsu

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

The application of text-to-speech (TTS) conversion has become widely used in recent years. Chinese TTS faces several unique difficulties. The most critical is caused by the lack of word delimiters in written Chinese. This means that Chinese word segmentation (CWS) must be the first step in Chinese TTS. Unfortunately, due to the ambiguous nature of word boundaries in Chinese, even the best CWS systems make serious segmentation errors. Incorrect sentence interpretation causes TTS errors, preventing TTS's wider use in applications such as automatic customer services or computer reader systems for the visually impaired. In this paper, we propose a novel method that exploits unlabeled internal data to reduce word segmentation errors without using external dictionaries. To demonstrate the generality of our method, we verify our system on the most widely recognized CWS evaluation tool--the SIGHAN bakeoff, which includes datasets in both traditional and simplified Chinese. These datasets are provided by four representative academies or industrial research institutes in HK, Taiwan, Mainland China, and the U.S. Our experimental results show that with only internal data and unlabeled test data, our approach reduces segmentation errors by an average of 15% compared to the traditional approach. Moreover, our approach achieves comparable performance to the best CWS systems that use external resources. Further analysis shows that our method has the potential to become more accurate as the amount of test data increases.

Original languageEnglish
Title of host publicationInternational Speech Communication Association - 8th Annual Conference of the International Speech Communication Association, Interspeech 2007
Pages2944-2947
Number of pages4
StatePublished - 2007
Event8th Annual Conference of the International Speech Communication Association, Interspeech 2007 - Antwerp, Belgium
Duration: 27 Aug 200731 Aug 2007

Publication series

NameInternational Speech Communication Association - 8th Annual Conference of the International Speech Communication Association, Interspeech 2007
Volume4

Conference

Conference8th Annual Conference of the International Speech Communication Association, Interspeech 2007
Country/TerritoryBelgium
CityAntwerp
Period27/08/0731/08/07

Keywords

  • Chinese word segmentation
  • Internal unlabeled data
  • Segmentation errors
  • Text-to-speech

Fingerprint

Dive into the research topics of 'Exploiting unlabeled internal data in conditional random fields to reduce word segmentation errors for Chinese texts'. Together they form a unique fingerprint.

Cite this