skip to main content skip to footer

On the Consequences of Some Test Rescoring Policies

Author(s):
Bradlow, Eric T.; Wainer, Howard
Publication Year:
1996
Report Number:
RR-96-36
Source:
ETS Research Report
Document Type:
Report
Page Count:
35
Subject/Key Words:
Constructed Responses, Interrater Reliability, Linear Models, Scoring, Variance Component Analysis

Abstract

When tests or portions of tests are scored subjectively by raters, a rescoring will yield a change in the ratings of some examinees. In a high stakes test with a fixed passing score a rescoring will result in the change of some pass/fail decisions. The number of changes depends on three things: (a) the reliability of the rating system, (b) the proportion of examinees that are expected to pass, and (c) the policy used to incorporate the rescore into the pass/fail decision. In this study, we provide a model that facilitates the evaluation of various rescoring strategies. We illustrate the use of this model for rescoring schemes that concatenate all ratings, old and new. The estimation of rescoring effects are computed by direct simulation. A further generalization of the basic model is also considered in which a test is comprised of a mixture of objectively and subjectively scored items. Last, we note that rescoring is merely a special case of the more general problem of the optimal allocation of judges and demonstrate an algorithm that accomplishes this.

Read More