skip to main content skip to footer

Remote Scoring of Essays

Breland, Hunter M.; Jones, Robert J.
Publication Year:
Report Number:
RR-88-04, CBR-88-03
ETS Research Report
Document Type:
Page Count:
Subject/Key Words:
College Board, Essay Tests, Evaluators, Holistic Evaluation, Scoring, Test Reliability, Test Validity, Writing Evaluation


Essays written by college freshmen on two different topics were scored first by readers working in a conference setting and second by another set of readers working in their own homes or offices. The conference readers were trained in the standard manner on the specific topics to be scored and were monitored by table leaders, as is done in standard scoring procedures. The remote readers received only written instructions in the mail, and there was no monitoring of their scoring. Analyses of the reliability and validity of the two scoring methods were then conducted. The comparisons conducted suggest some promise for remote scoring, especially if more sophisticated calibration procedures can be developed and if improved reader monitoring can be implemented, but with current procedures remote scoring is not as effective as conference scoring. Under certain circumstances, however, where cost or convenience may be relatively more important than reliability and validity, remote scoring may be preferable to conference scoring. (44pp.)

Read More