Analyzing Item Generation with Natural Language Processing Tools for the TOEIC Listening Test
- Yoon, Su-Youn; Lee, Chong Min; Houghton, Patrick; Lopez, Melissa; Sakano, Jennifer; Loukina, Anastassia; Krovetz, Robert; Lu, Chi; Madnani, Nitin
- Publication Year:
- Report Number:
- ETS Research Report
- Document Type:
- Page Count:
- Subject/Key Words:
- Automated Item Generation Automated Scoring and Natural Language Processing Surveys Test of English for International Communication (TOEIC) TOEIC Listening and Reading Tests User Study Vocabulary Assessments Word Similarity
The TOEIC Listening test includes items or tasks related to the global workplace and with a variety of authentic contexts. As the need for an ever larger number of test forms has increased, an important goal for the TOEIC Listening test has been to increase the efficiency of item generations by maintaining a large pool of items across a wide range of contexts has been an important goal for the TOEIC Listening test.
This study aimed to support this goal by creating a set of automated natural language processing tools and resources that support item generation, including:
1. an automated tool that retrieves appropriate real-world videos
2. a list of vocabulary tagged with established difficulty levels, and a tool that suggests words
3. an automated tool that suggests similar words or phrases to a given word
These tools and resources were designed to help item writers by providing initial ideas, authentic language, and support for adjusting the variety and complexity of vocabulary in their items.
To evaluate the impact of these tools and resources on the efficiency of item generation, a small number of item writers created TOEIC Listening items by means of these tools, which facilitated access to a greater variety of authentic real-world listening contexts. The tools appeared to help item writers increase the variety of topics covered by TOEIC Listening tasks.Ultimately, the regular use of these tools and resources may help strengthen the link between real-world workplace listening contexts and TOEIC Listening tasks. This stronger link will provide additional support for the validity of TOEIC Listening test score interpretations; specifically, that they align with listening situations in the workplace and in everyday life.
In this study, we developed assistive tools and resources to support TOEIC Listening test item generation. There has recently been an increased need for a large pool of items for these tests. This need has, in turn, inspired efforts to increase the efficiency of item generation while maintaining the quality of the created items. We aimed to address this challenge by creating a set of automated tools and resources that support item generation: an automated system that retrieves appropriate real-world videos, a list of vocabulary tagged with established difficulty levels, and a tool that suggests words and phrases that are similar in distribution to a given word (word similarity tool). These tools and resources were designed to help item writers by providing initial ideas, authentic language, and support for adjusting the variety and complexity of vocabulary in their items. To evaluate the impact of these resources on the efficiency of item generation, seven item writers created listening items using our tools. All tools were considered useful, and the word similarity tool in particular was rated the most useful. The tools are currently applied to English item generation for the TOEIC Listening test, but the method is generic and applicable to other languages.