skip to main content skip to footer

Reliability of Essay Rating and Score Adjustment

Author(s):
Longford, Nicholas T.
Publication Year:
1993
Report Number:
RR-93-52
Source:
ETS Research Report
Document Type:
Report
Page Count:
50
Subject/Key Words:
Essay Tests, Models, Reader Reliability, Scoring, Test Reliability

Abstract

(48pp.) A model-based approach to rater reliability for essays read by multiple readers is presented. Variation of rater severity (between-rater variation) and rater inconsistency (within-rater variation) is considered in presence of between-examinee variation. An additive variance component model is posited and the method of moments for its estimation described. The model involves no distributional assumptions. Minimum mean squared error estimators of examinees' true scores and readers' severities are derived. Model diagnostic procedures are an integral component of the approach. The methods are illustrated on data from standardized educational tests.

Read More