View in EDS PDF Full Text

Language identification of multilingual posts from Twitter: a case study.

Saved in:
Bibliographic Details
Title: Language identification of multilingual posts from Twitter: a case study.
Authors: Pla, Ferran1 fpla@dsic.upv.es, Hurtado, Lluís-F.1 lhurtado@dsic.upv.es
Source: Knowledge & Information Systems. Jun2017, Vol. 51 Issue 3, p965-989. 25p.
Subjects: Language identification (Computational linguistics), Multilingual computing, Twitter (Web resource), Support vector machines, Classification, Problem solving
Abstract: This paper describes a method for handling multi-class and multi-label classification problems based on the support vector machine formalism. This method has been applied to the language identification problem in Twitter. The system evaluation was performed mainly on a Twitter data set developed in the TweetLID workshop. This data set contains bilingual tweets written in the most commonly used Iberian languages (i.e., Spanish, Portuguese, Catalan, Basque, and Galician) as well as the English language. We address the following problems: (1) social media texts. We propose a suitable tokenization that processes the peculiarities of Twitter; (2) multilingual tweets. Since a tweet can belong to more than one language, we need to use a multi-class and multi-label classifier; (3) similar languages. We study the main confusions among similar languages; and (4) unbalanced classes. We propose threshold-based strategy to favor classes with less data. We have also studied the use of Wikipedia and the addition of new tweets in order to increase the training data set. Additionally, we have tested our system on Bergsma corpus, a collection of tweets in nine languages, focusing on confusable languages using the Cyrillic, Arabic, and Devanagari alphabets. To our knowledge, we obtained the best results published on the TweetLID data set and results that are in line with the best results published on Bergsma data set. [ABSTRACT FROM AUTHOR]
Copyright of Knowledge & Information Systems is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
FullText Links:
  – Type: pdflink
Text:
  Availability: 0
Header DbId: egs
DbLabel: Engineering Source
An: 122897014
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Language identification of multilingual posts from Twitter: a case study.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Pla%2C+Ferran%22">Pla, Ferran</searchLink><relatesTo>1</relatesTo><i> fpla@dsic.upv.es</i><br /><searchLink fieldCode="AR" term="%22Hurtado%2C+Lluís-F%2E%22">Hurtado, Lluís-F.</searchLink><relatesTo>1</relatesTo><i> lhurtado@dsic.upv.es</i>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22Knowledge+%26+Information+Systems%22">Knowledge & Information Systems</searchLink>. Jun2017, Vol. 51 Issue 3, p965-989. 25p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Language+identification+%28Computational+linguistics%29%22">Language identification (Computational linguistics)</searchLink><br /><searchLink fieldCode="DE" term="%22Multilingual+computing%22">Multilingual computing</searchLink><br /><searchLink fieldCode="DE" term="%22Twitter+%28Web+resource%29%22">Twitter (Web resource)</searchLink><br /><searchLink fieldCode="DE" term="%22Support+vector+machines%22">Support vector machines</searchLink><br /><searchLink fieldCode="DE" term="%22Classification%22">Classification</searchLink><br /><searchLink fieldCode="DE" term="%22Problem+solving%22">Problem solving</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This paper describes a method for handling multi-class and multi-label classification problems based on the support vector machine formalism. This method has been applied to the language identification problem in Twitter. The system evaluation was performed mainly on a Twitter data set developed in the TweetLID workshop. This data set contains bilingual tweets written in the most commonly used Iberian languages (i.e., Spanish, Portuguese, Catalan, Basque, and Galician) as well as the English language. We address the following problems: (1) social media texts. We propose a suitable tokenization that processes the peculiarities of Twitter; (2) multilingual tweets. Since a tweet can belong to more than one language, we need to use a multi-class and multi-label classifier; (3) similar languages. We study the main confusions among similar languages; and (4) unbalanced classes. We propose threshold-based strategy to favor classes with less data. We have also studied the use of Wikipedia and the addition of new tweets in order to increase the training data set. Additionally, we have tested our system on Bergsma corpus, a collection of tweets in nine languages, focusing on confusable languages using the Cyrillic, Arabic, and Devanagari alphabets. To our knowledge, we obtained the best results published on the TweetLID data set and results that are in line with the best results published on Bergsma data set. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of Knowledge & Information Systems is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=122897014
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s10115-016-0997-x
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 25
        StartPage: 965
    Subjects:
      – SubjectFull: Language identification (Computational linguistics)
        Type: general
      – SubjectFull: Multilingual computing
        Type: general
      – SubjectFull: Twitter (Web resource)
        Type: general
      – SubjectFull: Support vector machines
        Type: general
      – SubjectFull: Classification
        Type: general
      – SubjectFull: Problem solving
        Type: general
    Titles:
      – TitleFull: Language identification of multilingual posts from Twitter: a case study.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Pla, Ferran
      – PersonEntity:
          Name:
            NameFull: Hurtado, Lluís-F.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 06
              Text: Jun2017
              Type: published
              Y: 2017
          Identifiers:
            – Type: issn-print
              Value: 02191377
          Numbering:
            – Type: volume
              Value: 51
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: Knowledge & Information Systems
              Type: main
ResultId 1