skip to main content skip to footer

How Do Raters From India Perform in Scoring the TOEFL iBT Speaking Section and What Kind of Training Helps? TOEFL iBT

Author(s):
Xi, Xiaoming; Mollaun, Pamela
Publication Year:
2009
Report Number:
RR-09-31
Source:
ETS Research Report
Document Type:
Report
Page Count:
37
Subject/Key Words:
Test of English as a Foreign Language (TOEFL), Internet Based Testing (iBT), India, Speaking, Speech Scoring, Rater Background Characteristics, Non-Native Raters, Rater Bias

Abstract

This study investigated the scoring of the Test of English as a Foreign Language Internet-based Test (TOEFL iBT) Speaking section by bilingual or multilingual speakers of English and 1 or more Indian languages. We explored the extent to which raters from India, after being trained and certified, were able to score the Speaking section for TOEFL iBT examinees with mixed first language (L1) backgrounds, especially those speaking an Indian language, accurately and consistently. The effectiveness of a special training package designed for scoring Indian examinees was examined as well. A total of 26 trained and certified raters from India were randomly divided into 2 groups and participated in 2 on-site scoring sessions in Mumbai. In the first session, both groups received regular training for scoring the TOEFL iBT Speaking section, which was largely similar to that received by raters in North America. In the second scoring session, 1 group continued to receive the regular training while the second group was trained using a special training package. Rater feedback surveys were also given to the raters. It was found that with training similar to that which operational U.S.-based raters receive, the raters from India performed as well as the operational raters in scoring both Indian and non-Indian examinees. In addition, the special training helped the raters score Indian examinees more consistently, leading to increased score reliability estimates. It also boosted raters’ levels of confidence in scoring Indian examinees.

Read More