An Interesting Problem in the Estimation of Scoring Reliability

Author(s):

Livingston, Samuel A.

Publication Year:

2003

Report Number:

RM-03-02

Source:

ETS Research Memorandum

Document Type:

Report

Page Count:

7

Subject/Key Words:

Interrater Reliability, Scoring

Abstract

A performance assessment consisting of 10 separate exercises was scored with a randomized scoring procedure. All responses to each exercise were rated; then a randomly selected subset of the responses to each exercise received an independent second rating. Each second rating was averaged with the corresponding first rating before the scores were computed. This report presents a method for estimating the scoring reliability coefficient (inter-rater reliability) and the standard error of scoring of the resulting scores. The report concludes with a numerical example.

Read More

• Request Copy (specify title and report number, if any)
• http://www.ets.org/Media/Research/pdf/RM-03-02-Livingston.pdf