skip to main content skip to footer

The Impact of Using Synthetically Generated Listening Stimuli on Test-Taker Performance: A Case Study With Multiple-Choice, Single-Selection Items TOEFL L2 IRT

Author(s):
Choi, Ikkyu; Zu, Jiyun
Publication Year:
2022
Report Number:
RR-22-05
Source:
ETS Research Report
Document Type:
Report
Page Count:
16
Subject/Key Words:
TOEFL Essentials, Test of English as a Foreign Language (TOEFL), Speech Synthesis, Oral Communication, Listening Comprehension, Secondary Language (L2), Multiple Choice Items, Test-Taker Performance, Item Difficulty, Item Discrimination (Tests), Item Response Theory (IRT)

Abstract

Synthetically generated speech (SGS) has become an integral part of our oral communication in a wide variety of contexts. It can be generated instantly at a low cost and allows precise control over multiple aspects of output, all of which can be highly appealing to second language (L2) assessment developers who have traditionally relied upon human voice actors for recording audio materials. Nevertheless, SGS is not widely used in L2 assessments. One major concern in this use case lies in its potential impact on test-taker performance: Would the use of SGS (as opposed to using human voice actor recordings) change how test takers respond to an item? In this study, we investigated using SGS as stimuli for English L2 listening assessment items on test-taker performance. The data came from a pilot administration of multiple new task types and included 653 test takers’ responses to two versions of the same 13 items, differing only in terms of their listening stimuli: a version using human voice actor recordings and the other version with SGS files. Multifaceted comparisons between test takers’ responses across the two versions showed that the two versions elicited remarkably comparable performance. The comparability provides strong empirical evidence for the use of SGS as a viable alternative for human voice actor recordings in the immediate domain of L2 assessment as well as related domains such as learning material and research instrument development.

Read More