skip to main content skip to footer

Validation of Automated Scores of TOEFL iBT Tasks Against Nontest Indicators of Writing Ability TOEFL iBT

Author(s):
Weigle, Sara Cushing
Publication Year:
2011
Report Number:
RR-11-24
Source:
ETS Research Report
Document Type:
Report
Page Count:
63
Subject/Key Words:
Test of English as a Foreign Language (TOEFL), Internet Based Testing (iBT), Automated Scoring, Writing Assessment, Second Language, Validity, Electronic Essay Rater (E-rater)

Abstract

Automated scoring has the potential to dramatically reduce the time and costs associated with the assessment of complex skills such as writing, but its use must be validated against a variety of criteria for it to be accepted by test users and stakeholders. This study addresses two validity-related issues regarding the use of e-rater with the independent writing task on the TOEFL iBT (Internet-based test). First, relationships between automated scores of iBT tasks and nontest indicators of writing ability were examined. This was followed by exploration of prompt-related differences in automated scores of essays written by the same examinees. Correlations between both human and e-rater scores and nontest indicators were moderate but consistent, with few differences between e-rater and human rater scores. E-rater was more consistent across prompts than individual human raters, although there were differences in scores across prompts for the individual features used to generate total e-rater scores.

Read More