How ETS Scores the TOEIC Speaking and Writing Tests Responses TOEIC EFL ESL OSN
- Publication Year:
- Report Number:
- Document Type:
- Page Count:
- Subject/Key Words:
Test of English for International Communication (TOEIC),
English Language Proficiency,
English as a Foreign Language (EFL),
English as a Second Language (ESL),
English Language Skills,
Test Score Reliability,
Online Scoring Network (OSN)
SUMMARY: Typically, human raters are used to score Speaking and Writing tests because of their ability to evaluate a broader range of language performance than automated systems. This paper describes how ETS ensures the reliability and consistency of scores by human raters for TOEIC Speaking and Writing tests through training, certification, and systematic administrative and statistical monitoring procedures.
In order to qualify, raters must meet specific requirements, complete a systematic training course and pass a certification test. Prior to every rating session, raters must pass a calibration test which assesses their readiness to score on that day. During the scoring process itself, raters are monitored by scoring leaders which provide real-time feedback and support. The rating process uses ETS's patented online scoring network (OSN) system, which also helps ensure that each test taker receives multiple (up to nine), independent and anonymous ratings. This specific factor in the TOEIC scoring process minimizes the "Halo Effect" (bias) and "Human Error." The TOEIC scoring process as a whole helps generate more accurate and consistent test scores by minimizing all potential sources of error and bias.
ABSTRACT: This paper describes how ETS addresses the challenges presented by the use of human raters for the TOEIC Speaking and Writing tests. This paper is part of the Research Foundation for TOEIC: A Compendium of Studies, published by ETS in 2010.