skip to main content skip to footer

Sample-Size Requirements for Automated Essay Scoring AES

Author(s):
Haberman, Shelby J.; Sinharay, Sandip
Publication Year:
2008
Report Number:
RR-08-32
Source:
ETS Research Report
Document Type:
Report
Page Count:
43
Subject/Key Words:
Automated Essay Scoring (AES), Sample Size, Cumulative Logit Model (CLM)

Abstract

Sample-size requirements were considered for automated essay scoring in cases in which the automated essay score estimates the score provided by a human rater. Analysis considered both cases in which an essay prompt is examined in isolation and those in which a family of essay prompts is studied. In typical cases in which content analysis is not employed and in which the only object is to score individual essays to provide feedback to the examinee, it appears that several hundred essays are sufficient. For application of one model to a family of essays, fewer than 100 essays per prompt may often be adequate. The cumulative logit model was explored as a possible replacement of the linear regression model usually employed in automated essay scoring; the cumulative logit model performed somewhat better than did the linear regression model.

Read More