skip to main content skip to footer

Estimating Reliability for Tests With One Constructed-Response Item in a Section

Author(s):
Qu, Yanxuan; Sinharay, Sandip
Publication Year:
2024
Report Number:
RR-24-07
Source:
ETS Research Report
Document Type:
Report
Page Count:
18
Subject/Key Words:
Test Score Reliability, Constructed-Response Items, Internal Consistency, Attenuation Paradox, Alpha Coefficient, Tau-Equivalency, Stratification, Simulation Study, Dichotomous Items, Polytomous Items

Abstract

The goal of this paper is to find better ways to estimate the internal consistency reliability of scores on tests with a specific type of design that are often encountered in practice: tests with constructed-response items clustered into sections that are not parallel or tau-equivalent, and one of the sections has only one item. To estimate the reliability of scores on this kind of test, we propose a two-step approach (denoted as CA_STR) that first estimates the reliability of scores on the section with a single item using the correction for attenuation method and then estimates the reliability of scores on the whole test using the stratified coefficient alpha. We compared the CA_STR method with three other reliability estimation approaches under various conditions using both real and simulated data. We found that overall, the CA_STR method performed the best and it was easy to implement.

Read More