Chinese text segmentation: A hybrid approach using transductive learning and statistical association measures

Research output: Contribution to journalArticlepeer-review

17 Scopus citations

Abstract

Chinese text segmentation (CTS) is a fundamental step in building any Chinese or cross-language information retrieval system. This paper identifies and proposes solutions to two main challenges facing today's CTS systems: segmenting words longer than the context window and identifying words not derived from affixation or composition. Our methods exploit unlabeled data, making them scalable at little extra cost. To tackle the first problem, we use a transductive learning approach to automatically construct a dictionary, and then refine it by improving its test set coverage while reducing its over-fitting tendency. In addition, we incorporate frequency information to discriminate overlapping matching words. For the second problem, we employ statistical association measures non-parametrically through a natural but novel feature representation scheme. To demonstrate the generality of our approach, we verify our system on the most reputable CTS evaluation standard - the SIGHAN bakeoff, which contains datasets in both traditional and simplified Chinese. These datasets are provided by representative academic or industrial research institutes. The experimental results show that with only training data and unlabeled test data and with no external dictionaries, our approach effectively overcomes the above-mentioned problems and reduces segmentation errors by an average of 27.8% compared with the traditional approach. Notably, our approach improves the recall of new words, the most informative words, by 4.7% on average. Also, our approach outperforms the best SIGHAN CTS system, which requires many external resources. Additional analysis shows that our approach has the potential to gain accuracy as the test data increases.

Original languageEnglish
Pages (from-to)3553-3560
Number of pages8
JournalExpert Systems with Applications
Volume37
Issue number5
DOIs
StatePublished - May 2010

Keywords

  • Association measure
  • Chinese word segmentation
  • Transductive learning
  • Unlabeled data

Fingerprint

Dive into the research topics of 'Chinese text segmentation: A hybrid approach using transductive learning and statistical association measures'. Together they form a unique fingerprint.

Cite this