skip to main content skip to footer

Qualifying Essay Readers for an Online Scoring Network (OSN) OSN CAT

Author(s):
Powers, Donald E.; Kubota, Mel; Bentley, Jill; Farnum, Marisa; Swartz, Rich; Willard, Ann E.
Publication Year:
1998
Report Number:
RR-98-20
Source:
ETS Research Report
Document Type:
Report
Page Count:
21
Subject/Key Words:
Essay Tests, Scoring, Qualifications, Computer Assisted Testing, Online Systems, Constructed Response

Abstract

All study participants evaluated a set of essays before any training and another set after they had undergone standard training for scoring essays. The results showed that training did affect the accuracy with which readers (especially those who were previously inexperienced) scored essays. Moreover, after training, a significant proportion of inexperienced readers exhibited a level of accuracy that was commensurate with that shown by experienced readers. A comparison of the performances of experienced and inexperienced readers suggested that a defensible passing score of about 90% accuracy could be set on the reader certification test when the standard of accuracy is agreement within one point of previously established essay scores.

Read More