skip to main content skip to footer

Factors in Judgments of Writing Ability

Diederich, Paul B.; French, John Winslow, 1918-; Carlton, Sydell T.
Publication Year:
Report Number:
ETS Research Bulletin
Document Type:
Page Count:
Subject/Key Words:
Essay Tests, Evaluation Criteria, Factor Structure, Interrater Reliability, Writing Evaluation


The purpose of this study was to serve as a stepping stone toward closer agreement among judges of student writing at the point of admission to college by revealing common causes of disagreement. It was expected and found that more than half the variability in grades of a large number of judges on the same set of papers was due to "error" (random variation) or the idiosyncratic preferences of individual readers. In the variability that was not random or idiosyncratic, it was expected and found that there was a substantial core of common agreement on the general merit of the papers and that a small number of "schools of thought" would account for most of the systematic differences in grading standards. Factor analysis of correlations among the grades of 53 distinguished readers, representing six different fields, on 300 papers written by college freshmen of widely varying ability revealed just five such "schools of thought." It was not the purpose of this study to achieve a high degree of unanimity among the readers but to reveal the differences of opinion that prevail in uncontrolled grading--both in the academic community and in the educated public. To that end, the readers included college English teachers, social scientists, natural scientists, writers and editors, lawyers, and business executives. None the less, it was disturbing to find that 94% of the papers received either seven, eight, or nine of the nine possible grades; that no paper received less than five different grades; and that the median correlation between readers was .31. Readers in each field, however, agreed slightly better with the English teachers than with one another.

Read More