skip to main content skip to footer

Choice of Target Population Weights in Rater Comparability Scoring and Equating FE

Author(s):
Puhan, Gautam
Publication Year:
2013
Report Number:
RR-13-03
Source:
ETS Research Report
Document Type:
Report
Page Count:
17
Subject/Key Words:
Rater Comparability Scoring, Target Population, Tucker Equating, Frequency Estimation (FE) Equipercentile Equating

Abstract

The purpose of this study was to demonstrate that the choice of sample weights when defining the target population under poststratification equating can be a critical factor in determining the accuracy of the equating results under a unique equating scenario, known as rater comparability scoring and equating. The nature of data collection under rater comparability scoring is such that it results in a very high correlation between the anchor and total score in the new form but only a moderate correlation in the reference form. I demonstrated, using data collected under a rater comparability scoring situation, that this difference in the anchor-total correlation in the new and reference forms can have a predictable impact on the equating results based on different sample weights (i.e., the equating results are most accurate when the reference form sample is defined as the target population, least accurate when the new form sample is defined as the target population, and somewhere in the middle when the new and reference form samples are equally weighed when defining the target population).

Read More