skip to main content skip to footer

An Examination of the Link Between Rater Calibration Performance and Subsequent Scoring Accuracy in Graduate Record Examinations (GRE) Writing GRE

Author(s):
Ricker-Pedley, Kathryn L.
Publication Year:
2011
Report Number:
RR-11-03
Source:
ETS Research Report
Document Type:
Report
Page Count:
22
Subject/Key Words:
Rater Accuracy, Calibration, Graduate Record Examination (GRE), GRE Writing Assessment

Abstract

A pseudo-experimental study was conducted to examine the link between rater accuracy calibration performances and subsequent accuracy during operational scoring. The study asked 45 raters to score a 75-response calibration set and then a 100-response (operational) set of responses from a retired Graduate Record Examinations (GRE) writing prompt that had been vetted to determine correct scores prior to the study. The study found a positive relationship between calibration accuracy and scoring accuracy. Results suggest that a longer calibration test is more reliable and has a stronger correlation with operational scoring exact agreement; however, even tests as short as 10 responses had a reasonable correlation with operational accuracy, and the shorter test classified raters accurately 87% of the time when compared to the 75-response criterion. The additional requirement of no discrepant scores during calibration had better correlation with exact agreement, but much poorer classification accuracy in the shorter tests. This research suggests that calibration is useful for screening raters in its current form, but that further investigation of passing standards and of less controlled operational scoring conditions is required.

Read More