skip to main content skip to footer

Reliability of Reading of the College Board English Composition Test, December 1963

Author(s):
Coffman, William E.; McConville, Carolyn B.; Myers, Albert E.
Publication Year:
1964
Report Number:
RB-64-42, RDR-64-65, No. 03
Source:
ETS Research Bulletin
Document Type:
Report
Page Count:
24
Subject/Key Words:
College Board, College Entrance Examinations, English Composition Tests, Essay Tests, Interrater Reliability, Scoring, Writing Evaluation

Abstract

In December 1963, the College Board English Composition Test included a 20-minute essay. Approximately 80,000 candidates wrote the examination, and the essays were read by 145 readers during the week following the administration. The purpose of this study was to assess the reliability of the reading, based on a sample of 125 papers. An analysis of variance procedure was used to assess the reliability of the judgments made by the readers. The reader reliability of the sum of two readings (the score used in reporting results) was .577, a figure that was consistent with those obtained in previous experimental investigations and indicated that there was no loss in reliability in moving from experimental reading of fewer than 1,000 papers to operational reading of more than 80,000 papers. Single judgment reliabilities were .41 on the average, but this average was reduced by the very low .26 that was found for readings made on the last day. It was concluded that this drop probably represented an "end effect" in which the readers relaxed their vigilance as they approached the end of the demanding reading period. Other evidence suggested that the readers were making global judgments, as directed.

Read More