skip to main content skip to footer

Evaluating the Stability of Test Score Means for the TOEIC Speaking and Writing Tests TOEIC ANOVA

Author(s):
Qu, Yanxuan; Huo, Yan; Chan, Eric; Shotts, Matthew
Publication Year:
2017
Report Number:
RR-17-50
Source:
ETS Research Report
Document Type:
Report
Page Count:
10
Subject/Key Words:
Score Stability, Test of English for International Communication (TOEIC), Speaking Assessments, Writing Tests, Test-Taker Performance, Harmonic Regression, Analysis of Variance (ANOVA), Seasonality, Scale Stability, Quality Control, Linear Regression

Abstract

For educational tests, it is critical to maintain consistency of score scales and to understand the sources of variation in score means over time. This practice helps to ensure that interpretations about test takers' abilities are comparable from one administration (or one form) to another. This study examines the consistency of reported scores for the TOEIC Speaking and Writing tests using statistical procedures. Specifically, the stability of the TOEIC Speaking score means from 431 forms administered in a 3-year period was evaluated using harmonic regression, and the stability of TOEIC Writing score means from 66 forms administered in a 3-year period was evaluated using analysis of variance. Results indicated that the fluctuations in the TOEIC Speaking or Writing score means mainly reflect changes in test takers' overall English speaking or writing ability levels instead of score inaccuracies. For both speaking and writing test scores, a large proportion of the variation in score means can be explained by seasonality (the rise or fall of score means associated with specific times of the year) and test takers' demographic information, which have been shown to be related to test-taker ability. As a result, this finding provides evidence for the consistency of the TOEIC Speaking and Writing score scales across forms.

Read More