View in EDS

Analyzing Item Generation with Natural Language Processing Tools for the 'TOEIC'® Listening Test. Research Report. ETS RR-17-52

Saved in:
Bibliographic Details
Title: Analyzing Item Generation with Natural Language Processing Tools for the 'TOEIC'® Listening Test. Research Report. ETS RR-17-52
Language: English
Authors: Yoon, Su-Youn, Lee, Chong Min, Houghton, Patrick, Lopez, Melissa, Sakano, Jennifer, Loukina, Anastasia, Krovetz, Bob, Lu, Chi, Madani, Nitin
Source: ETS Research Report Series. Dec 2017.
Availability: Educational Testing Service. Rosedale Road, MS19-R Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/research/policy_research_reports/ets
Peer Reviewed: Y
Page Count: 11
Publication Date: 2017
Document Type: Journal Articles
Reports - Research
Descriptors: Natural Language Processing, Language Tests, Item Banks, Pilot Projects, English (Second Language), Second Language Learning, Teaching Methods, Listening Comprehension Tests, Surveys, Interviews, Regression (Statistics)
Assessment and Survey Identifiers: Test of English for International Communication
ISSN: 2330-8516
Abstract: In this study, we developed assistive tools and resources to support TOEIC® Listening test item generation. There has recently been an increased need for a large pool of items for these tests. This need has, in turn, inspired efforts to increase the efficiency of item generation while maintaining the quality of the created items. We aimed to address this challenge by creating a set of automated tools and resources that support item generation: an automated system that retrieves appropriate real-world videos, a list of vocabulary tagged with established difficulty levels, and a tool that suggests words and phrases that are similar in distribution to a given word (word similarity tool). These tools and resources were designed to help item writers by providing initial ideas, authentic language, and support for adjusting the variety and complexity of vocabulary in their items. To evaluate the impact of these resources on the efficiency of item generation, seven item writers created listening items using our tools. All tools were considered useful, and the word similarity tool in particular was rated the most useful. The tools are currently applied to English item generation for the TOEIC Listening test, but the method is generic and applicable to other languages.
Abstractor: As Provided
Number of References: 14
Entry Date: 2018
Accession Number: EJ1168427
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1168427
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: EJ1168427
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Analyzing Item Generation with Natural Language Processing Tools for the 'TOEIC'® Listening Test. Research Report. ETS RR-17-52
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Yoon%2C+Su-Youn%22">Yoon, Su-Youn</searchLink><br /><searchLink fieldCode="AR" term="%22Lee%2C+Chong+Min%22">Lee, Chong Min</searchLink><br /><searchLink fieldCode="AR" term="%22Houghton%2C+Patrick%22">Houghton, Patrick</searchLink><br /><searchLink fieldCode="AR" term="%22Lopez%2C+Melissa%22">Lopez, Melissa</searchLink><br /><searchLink fieldCode="AR" term="%22Sakano%2C+Jennifer%22">Sakano, Jennifer</searchLink><br /><searchLink fieldCode="AR" term="%22Loukina%2C+Anastasia%22">Loukina, Anastasia</searchLink><br /><searchLink fieldCode="AR" term="%22Krovetz%2C+Bob%22">Krovetz, Bob</searchLink><br /><searchLink fieldCode="AR" term="%22Lu%2C+Chi%22">Lu, Chi</searchLink><br /><searchLink fieldCode="AR" term="%22Madani%2C+Nitin%22">Madani, Nitin</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22ETS+Research+Report+Series%22"><i>ETS Research Report Series</i></searchLink>. Dec 2017.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Educational Testing Service. Rosedale Road, MS19-R Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/research/policy_research_reports/ets
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 11
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2017
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Tests%22">Language Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Banks%22">Item Banks</searchLink><br /><searchLink fieldCode="DE" term="%22Pilot+Projects%22">Pilot Projects</searchLink><br /><searchLink fieldCode="DE" term="%22English+%28Second+Language%29%22">English (Second Language)</searchLink><br /><searchLink fieldCode="DE" term="%22Second+Language+Learning%22">Second Language Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Teaching+Methods%22">Teaching Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Listening+Comprehension+Tests%22">Listening Comprehension Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Surveys%22">Surveys</searchLink><br /><searchLink fieldCode="DE" term="%22Interviews%22">Interviews</searchLink><br /><searchLink fieldCode="DE" term="%22Regression+%28Statistics%29%22">Regression (Statistics)</searchLink>
– Name: SubjectThesaurus
  Label: Assessment and Survey Identifiers
  Group: Su
  Data: <searchLink fieldCode="SU" term="%22Test+of+English+for+International+Communication%22">Test of English for International Communication</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 2330-8516
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: In this study, we developed assistive tools and resources to support TOEIC® Listening test item generation. There has recently been an increased need for a large pool of items for these tests. This need has, in turn, inspired efforts to increase the efficiency of item generation while maintaining the quality of the created items. We aimed to address this challenge by creating a set of automated tools and resources that support item generation: an automated system that retrieves appropriate real-world videos, a list of vocabulary tagged with established difficulty levels, and a tool that suggests words and phrases that are similar in distribution to a given word (word similarity tool). These tools and resources were designed to help item writers by providing initial ideas, authentic language, and support for adjusting the variety and complexity of vocabulary in their items. To evaluate the impact of these resources on the efficiency of item generation, seven item writers created listening items using our tools. All tools were considered useful, and the word similarity tool in particular was rated the most useful. The tools are currently applied to English item generation for the TOEIC Listening test, but the method is generic and applicable to other languages.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Ref
  Label: Number of References
  Group: RefInfo
  Data: 14
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2018
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1168427
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1168427
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 11
    Subjects:
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Language Tests
        Type: general
      – SubjectFull: Item Banks
        Type: general
      – SubjectFull: Pilot Projects
        Type: general
      – SubjectFull: English (Second Language)
        Type: general
      – SubjectFull: Second Language Learning
        Type: general
      – SubjectFull: Teaching Methods
        Type: general
      – SubjectFull: Listening Comprehension Tests
        Type: general
      – SubjectFull: Surveys
        Type: general
      – SubjectFull: Interviews
        Type: general
      – SubjectFull: Regression (Statistics)
        Type: general
      – SubjectFull: Test of English for International Communication
        Type: general
    Titles:
      – TitleFull: Analyzing Item Generation with Natural Language Processing Tools for the 'TOEIC'® Listening Test. Research Report. ETS RR-17-52
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Yoon, Su-Youn
      – PersonEntity:
          Name:
            NameFull: Lee, Chong Min
      – PersonEntity:
          Name:
            NameFull: Houghton, Patrick
      – PersonEntity:
          Name:
            NameFull: Lopez, Melissa
      – PersonEntity:
          Name:
            NameFull: Sakano, Jennifer
      – PersonEntity:
          Name:
            NameFull: Loukina, Anastasia
      – PersonEntity:
          Name:
            NameFull: Krovetz, Bob
      – PersonEntity:
          Name:
            NameFull: Lu, Chi
      – PersonEntity:
          Name:
            NameFull: Madani, Nitin
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Type: published
              Y: 2017
          Identifiers:
            – Type: issn-electronic
              Value: 2330-8516
          Titles:
            – TitleFull: ETS Research Report Series
              Type: main
ResultId 1