The Consistency of TOEIC Speaking Scores Across Ratings and Tasks
- Schmidgall, Jonathan
- Publication Year:
- Report Number:
- ETS Research Report
- Document Type:
- Page Count:
- Subject/Key Words:
- Generalizability Theory Human Raters Rater Consistency Rater Reliability Scale Scores Test of English for International Communication (TOEIC) TOEIC Speaking Test Validation
This study examines the consistency of TOEIC Speaking scores. The analysis uses a methodology based on generalizability theory, which allows researchers to examine the degree to which aspects of the testing procedure (i.e., raters, tasks) influence scores. The results contribute evidence to support claims that TOEIC Speaking scores are consistent.
This report briefly reviews the design and scoring procedure for the TOEIC Speaking test and summarizes existing evidence about the consistency of TOEIC Speaking test scores. It then describes several analyses conducted using generalizability theory to provide additional information about the consistency of scores across different aspects of the scoring procedure. At the individual task level, most of the variation in scores was explained by individual ability as opposed to differences between ratings. At the claim level, variation in scores was also mostly explained by individual ability rather than differences between tasks or ratings. Most importantly, variance in total scores was largely explained by individual ability rather than differences between ratings. The results of these analyses provide more robust information and support claims about consistency of TOEIC Speaking scores.