angle-up angle-right angle-down angle-left close user menu open menu closed search globe bars phone store

How ETS Scores the TOEIC Speaking and Writing Tests Responses

Author(s):
Everson, Philip; Hines, Susan
Publication Year:
2010
Report Number:
TC-10-08
Source:
TOEIC Compendium
Document Type:
Report
Page Count:
9
Subject/Key Words:
English as a Foreign Language (EFL) English as a Second Language (ESL) English Language Proficiency English Language Skills Human Raters Online Scoring Network (OSN) Self-Assessment Speaking Skills Test of English for International Communication (TOEIC) Test Score Reliability Test-Taker Performance Validity Writing Skills

Abstract

SUMMARY: Typically, human raters are used to score Speaking and Writing tests because of their ability to evaluate a broader range of language performance than automated systems. This paper describes how ETS ensures the reliability and consistency of scores by human raters for TOEIC Speaking and Writing tests through training, certification, and systematic administrative and statistical monitoring procedures. In order to qualify, raters must meet specific requirements, complete a systematic training course and pass a certification test. Prior to every rating session, raters must pass a calibration test which assesses their readiness to score on that day. During the scoring process itself, raters are monitored by scoring leaders which provide real-time feedback and support. The rating process uses ETS's patented online scoring network (OSN) system, which also helps ensure that each test taker receives multiple (up to nine), independent and anonymous ratings. This specific factor in the TOEIC scoring process minimizes the "Halo Effect" (bias) and "Human Error." The TOEIC scoring process as a whole helps generate more accurate and consistent test scores by minimizing all potential sources of error and bias. ABSTRACT: This paper describes how ETS addresses the challenges presented by the use of human raters for the TOEIC Speaking and Writing tests. This paper is part of the Research Foundation for TOEIC: A Compendium of Studies, published by ETS in 2010.

Read More