skip to main content skip to footer

Statistical Analyses for the Expanded TOEIC Speaking Test TOEIC

Author(s):
Qu, Yanxuan; Cid, Jaime; Chan, Eric; Huo, Yan
Publication Year:
2017
Report Number:
RM-17-03
Source:
ETS Research Memorandum
Document Type:
Report
Page Count:
20
Subject/Key Words:
Test of English for International Communication (TOEIC), TOEIC Speaking Test, Item Formats, Analysis of Covariance (ANCOVA), Item Difficulty, Test Reliability, Pilot Study

Abstract

Testing programs should periodically review their assessments to ensure that their test items or tasks are well aligned with real-world activities. For this reason, to better support communicative language learning and to discourage the use of memorization and other test-taking strategies, Educational Testing Service (ETS) expanded the existing format of some items of the TOEIC Speaking test in May 2015. It is important to ensure that the new expanded item formats are comparable to the existing formats. In this paper, we report the results of a pilot study conducted in November 2013 to evaluate the comparability of items with new and existing formats in terms of difficulty, score consistency, and overall test reliability. We also summarize the operational trends observed after the implementation of the expanded item formats. The results of the pilot study suggest that even though modifications to existing item formats had a slight effect on the difficulty of items, as some items were more difficult and others were less difficult, the effects observed were within the range of variation typically observed across different forms of the test. Further monitoring of the difficulties of the new item formats based on operational testing results also indicates that items with the new formats have performed similarly to items with existing formats. This report shows that the expansion in the TOEIC Speaking item formats did not have any significant undesirable effects on item difficulty or test score reliability, indicating that with the new, more authentic tasks, the TOEIC Speaking test scores remain consistent and reliable.

Read More