Analyzing Item Generation with Natural Language Processing Tools for the 'TOEIC'® Listening Test. Research Report. ETS RR-17-52
Saved in:
| Title: | Analyzing Item Generation with Natural Language Processing Tools for the 'TOEIC'® Listening Test. Research Report. ETS RR-17-52 |
|---|---|
| Language: | English |
| Authors: | Yoon, Su-Youn, Lee, Chong Min, Houghton, Patrick, Lopez, Melissa, Sakano, Jennifer, Loukina, Anastasia, Krovetz, Bob, Lu, Chi, Madani, Nitin |
| Source: | ETS Research Report Series. Dec 2017. |
| Availability: | Educational Testing Service. Rosedale Road, MS19-R Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/research/policy_research_reports/ets |
| Peer Reviewed: | Y |
| Page Count: | 11 |
| Publication Date: | 2017 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Natural Language Processing, Language Tests, Item Banks, Pilot Projects, English (Second Language), Second Language Learning, Teaching Methods, Listening Comprehension Tests, Surveys, Interviews, Regression (Statistics) |
| Assessment and Survey Identifiers: | Test of English for International Communication |
| ISSN: | 2330-8516 |
| Abstract: | In this study, we developed assistive tools and resources to support TOEIC® Listening test item generation. There has recently been an increased need for a large pool of items for these tests. This need has, in turn, inspired efforts to increase the efficiency of item generation while maintaining the quality of the created items. We aimed to address this challenge by creating a set of automated tools and resources that support item generation: an automated system that retrieves appropriate real-world videos, a list of vocabulary tagged with established difficulty levels, and a tool that suggests words and phrases that are similar in distribution to a given word (word similarity tool). These tools and resources were designed to help item writers by providing initial ideas, authentic language, and support for adjusting the variety and complexity of vocabulary in their items. To evaluate the impact of these resources on the efficiency of item generation, seven item writers created listening items using our tools. All tools were considered useful, and the word similarity tool in particular was rated the most useful. The tools are currently applied to English item generation for the TOEIC Listening test, but the method is generic and applicable to other languages. |
| Abstractor: | As Provided |
| Number of References: | 14 |
| Entry Date: | 2018 |
| Accession Number: | EJ1168427 |
| Database: | ERIC |
| Abstract: | In this study, we developed assistive tools and resources to support TOEIC® Listening test item generation. There has recently been an increased need for a large pool of items for these tests. This need has, in turn, inspired efforts to increase the efficiency of item generation while maintaining the quality of the created items. We aimed to address this challenge by creating a set of automated tools and resources that support item generation: an automated system that retrieves appropriate real-world videos, a list of vocabulary tagged with established difficulty levels, and a tool that suggests words and phrases that are similar in distribution to a given word (word similarity tool). These tools and resources were designed to help item writers by providing initial ideas, authentic language, and support for adjusting the variety and complexity of vocabulary in their items. To evaluate the impact of these resources on the efficiency of item generation, seven item writers created listening items using our tools. All tools were considered useful, and the word similarity tool in particular was rated the most useful. The tools are currently applied to English item generation for the TOEIC Listening test, but the method is generic and applicable to other languages. |
|---|---|
| ISSN: | 2330-8516 |
