View in EDS HTML Full Text PDF Full Text

Cross-language transfer of semantic annotation via targeted crowdsourcing: task design and evaluation.

Saved in:
Bibliographic Details
Title: Cross-language transfer of semantic annotation via targeted crowdsourcing: task design and evaluation.
Authors: Stepanov, Evgeny A.1 evgeny.stepanov@unitn.it, Chowdhury, Shammur Absar1 shammar.chowdhury@unitn.it, Bayer, Ali Orkan1 aliorkan.bayer@unitn.it, Ghosh, Arindam1 arindam.ghosh@unitn.it, Klasinas, Ioannis2 iklasinas@isc.tuc.gr, Calvo, Marcos3 marcoscalvo@google.com, Sanchis, Emilio4 esanchis@dsic.upv.es, Riccardi, Giuseppe1 giuseppe.riccardi@unitn.it
Source: Language Resources & Evaluation. Mar2018, Vol. 52 Issue 1, p341-364. 24p.
Subjects: Semantics, Multilingual computing, Crowdsourcing, Annotations, English language
Abstract: Modern data-driven spoken language systems (SLS) require manual semantic annotation for training spoken language understanding parsers. Multilingual porting of SLS demands significant manual effort and language resources, as this manual annotation has to be replicated. Crowdsourcing is an accessible and cost-effective alternative to traditional methods of collecting and annotating data. The application of crowdsourcing to simple tasks has been well investigated. However, complex tasks, like cross-language semantic annotation transfer, may generate low judgment agreement and/or poor performance. The most serious issue in cross-language porting is the absence of reference annotations in the target language; thus, crowd quality control and the evaluation of the collected annotations is difficult. In this paper we investigate targeted crowdsourcing for semantic annotation transfer that delegates to crowds a complex task such as segmenting and labeling of concepts taken from a domain ontology; and evaluation using source language annotation. To test the applicability and effectiveness of the crowdsourced annotation transfer we have considered the case of close and distant language pairs: Italian–Spanish and Italian–Greek. The corpora annotated via crowdsourcing are evaluated against source and target language expert annotations. We demonstrate that the two evaluation references (source and target) highly correlate with each other; thus, drastically reduce the need for the target language reference annotations. [ABSTRACT FROM AUTHOR]
Copyright of Language Resources & Evaluation is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
Text:
  Availability: 1
Header DbId: egs
DbLabel: Engineering Source
An: 127930801
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Cross-language transfer of semantic annotation via targeted crowdsourcing: task design and evaluation.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Stepanov%2C+Evgeny+A%2E%22">Stepanov, Evgeny A.</searchLink><relatesTo>1</relatesTo><i> evgeny.stepanov@unitn.it</i><br /><searchLink fieldCode="AR" term="%22Chowdhury%2C+Shammur+Absar%22">Chowdhury, Shammur Absar</searchLink><relatesTo>1</relatesTo><i> shammar.chowdhury@unitn.it</i><br /><searchLink fieldCode="AR" term="%22Bayer%2C+Ali+Orkan%22">Bayer, Ali Orkan</searchLink><relatesTo>1</relatesTo><i> aliorkan.bayer@unitn.it</i><br /><searchLink fieldCode="AR" term="%22Ghosh%2C+Arindam%22">Ghosh, Arindam</searchLink><relatesTo>1</relatesTo><i> arindam.ghosh@unitn.it</i><br /><searchLink fieldCode="AR" term="%22Klasinas%2C+Ioannis%22">Klasinas, Ioannis</searchLink><relatesTo>2</relatesTo><i> iklasinas@isc.tuc.gr</i><br /><searchLink fieldCode="AR" term="%22Calvo%2C+Marcos%22">Calvo, Marcos</searchLink><relatesTo>3</relatesTo><i> marcoscalvo@google.com</i><br /><searchLink fieldCode="AR" term="%22Sanchis%2C+Emilio%22">Sanchis, Emilio</searchLink><relatesTo>4</relatesTo><i> esanchis@dsic.upv.es</i><br /><searchLink fieldCode="AR" term="%22Riccardi%2C+Giuseppe%22">Riccardi, Giuseppe</searchLink><relatesTo>1</relatesTo><i> giuseppe.riccardi@unitn.it</i>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22Language+Resources+%26+Evaluation%22">Language Resources & Evaluation</searchLink>. Mar2018, Vol. 52 Issue 1, p341-364. 24p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Semantics%22">Semantics</searchLink><br /><searchLink fieldCode="DE" term="%22Multilingual+computing%22">Multilingual computing</searchLink><br /><searchLink fieldCode="DE" term="%22Crowdsourcing%22">Crowdsourcing</searchLink><br /><searchLink fieldCode="DE" term="%22Annotations%22">Annotations</searchLink><br /><searchLink fieldCode="DE" term="%22English+language%22">English language</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Modern data-driven spoken language systems (SLS) require manual semantic annotation for training spoken language understanding parsers. Multilingual porting of SLS demands significant manual effort and language resources, as this manual annotation has to be replicated. Crowdsourcing is an accessible and cost-effective alternative to traditional methods of collecting and annotating data. The application of crowdsourcing to simple tasks has been well investigated. However, complex tasks, like cross-language semantic annotation transfer, may generate low judgment agreement and/or poor performance. The most serious issue in cross-language porting is the absence of reference annotations in the target language; thus, crowd quality control and the evaluation of the collected annotations is difficult. In this paper we investigate <italic>targeted</italic> crowdsourcing for semantic annotation transfer that delegates to crowds a complex task such as segmenting and labeling of concepts taken from a domain ontology; and evaluation using source language annotation. To test the applicability and effectiveness of the crowdsourced annotation transfer we have considered the case of close and distant language pairs: Italian–Spanish and Italian–Greek. The corpora annotated via crowdsourcing are evaluated against source and target language expert annotations. We demonstrate that the two evaluation references (source and target) highly correlate with each other; thus, drastically reduce the need for the target language reference annotations. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of Language Resources & Evaluation is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=127930801
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s10579-017-9396-5
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 24
        StartPage: 341
    Subjects:
      – SubjectFull: Semantics
        Type: general
      – SubjectFull: Multilingual computing
        Type: general
      – SubjectFull: Crowdsourcing
        Type: general
      – SubjectFull: Annotations
        Type: general
      – SubjectFull: English language
        Type: general
    Titles:
      – TitleFull: Cross-language transfer of semantic annotation via targeted crowdsourcing: task design and evaluation.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Stepanov, Evgeny A.
      – PersonEntity:
          Name:
            NameFull: Chowdhury, Shammur Absar
      – PersonEntity:
          Name:
            NameFull: Bayer, Ali Orkan
      – PersonEntity:
          Name:
            NameFull: Ghosh, Arindam
      – PersonEntity:
          Name:
            NameFull: Klasinas, Ioannis
      – PersonEntity:
          Name:
            NameFull: Calvo, Marcos
      – PersonEntity:
          Name:
            NameFull: Sanchis, Emilio
      – PersonEntity:
          Name:
            NameFull: Riccardi, Giuseppe
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 03
              Text: Mar2018
              Type: published
              Y: 2018
          Identifiers:
            – Type: issn-print
              Value: 1574020X
          Numbering:
            – Type: volume
              Value: 52
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Language Resources & Evaluation
              Type: main
ResultId 1