skip to main content skip to footer

Adjustment for Reader Rating Behavior in the Test of Written English TWE

Author(s):
Longford, Nicholas T.
Publication Year:
1996
Report Number:
RR-95-39
Source:
ETS Research Report
Document Type:
Report
Page Count:
46
Subject/Key Words:
Essay Tests, Grading, Interrater Reliability, Scores, Test of Written English (TWE)

Abstract

This report evaluates the impact of a potential scheme for score adjustment using data from the administrations of the TWE® (Test of Written English™) in 1994. It is shown that, assuming non-informative assignment of readers to essays, the adjustment due to reader differences reduces the mean squared error for all essays except those graded by readers with small work loads. The quality of the rating process, as described by the variances due to true scores, severity, and inconsistency, as well as the distribution of work loads, are similar across the administrations. This would make a reliable prediction of the optimal score adjustment in future administrations possible. Two approximations to the optimal adjustment are proposed.

Read More