View in EDS PDF Full Text

Determination of Context Window Size.

Saved in:
Bibliographic Details
Title: Determination of Context Window Size.
Authors: Hung, Kei Yuen, Luk, Robert, Yeung, Daniel, Chung, Korris, Shu, Wenhuo
Source: International Journal of Computer Processing of Oriental Languages. Mar2001, Vol. 14 Issue 1, p71. 10p.
Subjects: Windows (Graphical user interfaces), Chinese character sets (Data processing)
Abstract: Context windows are important for a variety of natural language analysis and processing. A trade-off exists between the task performance and the size of the context. Lucassen and Mercer used mutual information to determine the size of the context for English text. We apply the same technique to determine the Context window size for Chinese text. In addition, we use the association score, proposed by Church. The association score is directly related to the prediction ability of units in the context. To reduce the effects of spurious associations, the association score values at the N% quartile is used, instead of the maximum, and the association score derived from low frequency occurrences (i.e. < 5) are discarded. A window size of 9 characters was found to be large enough for most associations between characters themselves, and between words themselves. An alternative approach using the (nonparametric) lambda statistic L[sub B] is examined, which overcomes spurious association problems and the averaging effect of mutual information. We conclude that the statistic is more suitable for exhaustive contextual models (e.g. variable Ngram models) whereas the association score is more suitable for non-exhaustive contextual models (e.g. identification of collocation). [ABSTRACT FROM AUTHOR]
Copyright of International Journal of Computer Processing of Oriental Languages is the property of World Scientific Publishing Company and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
FullText Links:
  – Type: pdflink
Text:
  Availability: 0
Header DbId: egs
DbLabel: Engineering Source
An: 9278357
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Determination of Context Window Size.
– Name: Author
  Label: Authors
  Group: Au
  Data: &lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Hung%2C+Kei+Yuen%22&quot;&gt;Hung, Kei Yuen&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Luk%2C+Robert%22&quot;&gt;Luk, Robert&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Yeung%2C+Daniel%22&quot;&gt;Yeung, Daniel&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Chung%2C+Korris%22&quot;&gt;Chung, Korris&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Shu%2C+Wenhuo%22&quot;&gt;Shu, Wenhuo&lt;/searchLink&gt;
– Name: TitleSource
  Label: Source
  Group: Src
  Data: &lt;searchLink fieldCode=&quot;JN&quot; term=&quot;%22International+Journal+of+Computer+Processing+of+Oriental+Languages%22&quot;&gt;International Journal of Computer Processing of Oriental Languages&lt;/searchLink&gt;. Mar2001, Vol. 14 Issue 1, p71. 10p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: &lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Windows+%28Graphical+user+interfaces%29%22&quot;&gt;Windows (Graphical user interfaces)&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Chinese+character+sets+%28Data+processing%29%22&quot;&gt;Chinese character sets (Data processing)&lt;/searchLink&gt;
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Context windows are important for a variety of natural language analysis and processing. A trade-off exists between the task performance and the size of the context. Lucassen and Mercer used mutual information to determine the size of the context for English text. We apply the same technique to determine the Context window size for Chinese text. In addition, we use the association score, proposed by Church. The association score is directly related to the prediction ability of units in the context. To reduce the effects of spurious associations, the association score values at the N% quartile is used, instead of the maximum, and the association score derived from low frequency occurrences (i.e. &lt; 5) are discarded. A window size of 9 characters was found to be large enough for most associations between characters themselves, and between words themselves. An alternative approach using the (nonparametric) lambda statistic L[sub B] is examined, which overcomes spurious association problems and the averaging effect of mutual information. We conclude that the statistic is more suitable for exhaustive contextual models (e.g. variable Ngram models) whereas the association score is more suitable for non-exhaustive contextual models (e.g. identification of collocation). [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: &lt;i&gt;Copyright of International Journal of Computer Processing of Oriental Languages is the property of World Scientific Publishing Company and its content may not be copied or emailed to multiple sites without the copyright holder&#39;s express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.&lt;/i&gt; (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=9278357
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 10
        StartPage: 71
    Subjects:
      – SubjectFull: Windows (Graphical user interfaces)
        Type: general
      – SubjectFull: Chinese character sets (Data processing)
        Type: general
    Titles:
      – TitleFull: Determination of Context Window Size.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Hung, Kei Yuen
      – PersonEntity:
          Name:
            NameFull: Luk, Robert
      – PersonEntity:
          Name:
            NameFull: Yeung, Daniel
      – PersonEntity:
          Name:
            NameFull: Chung, Korris
      – PersonEntity:
          Name:
            NameFull: Shu, Wenhuo
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 03
              Text: Mar2001
              Type: published
              Y: 2001
          Identifiers:
            – Type: issn-print
              Value: 02194279
          Numbering:
            – Type: volume
              Value: 14
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: International Journal of Computer Processing of Oriental Languages
              Type: main
ResultId 1