skip to main content skip to footer

Accuracy of a Classical Test Theory–Based Procedure for Estimating the Reliability of a Multistage Test MST CTT IRT

Author(s):
Kim, Sooyeon; Livingston, Samuel A.
Publication Year:
2017
Report Number:
RR-17-02
Source:
ETS Research Report
Document Type:
Report
Page Count:
11
Subject/Key Words:
Multistage Testing (MST), Classical Test Theory (CTT), Item Response Theory (IRT), Computer Adaptive Testing, Simulation-Based Assessment, Test Reliability, Estimation Theory

Abstract

The purpose of this simulation study was to assess the accuracy of a classical test theory (CTT)–based procedure for estimating the alternate-forms reliability of scores on a multistage test (MST) having 3 stages. We generated item difficulty and discrimination parameters for 10 parallel, nonoverlapping forms of the complete 3-stage test and ability parameters for a population of 30,000 simulated test takers. Using these parameters to generate item responses, we ran each of the 30,000 simulated test takers through each of the 10 simulated forms of the 3-stage test and computed the correlation of the scores on each pair of simulated test forms. We then computed the CTT estimate of the reliability, in the full population of 30,000 simulated test takers, of the total scaled scores resulting from the multistage testing procedure. We computed the estimate separately from the simulated responses to each of the 10 simulated forms of the 3-stage test. The reliability estimates from each simulated form of the MST differed by less than .005 from the average of the correlations between scores on that form and on the other 9 simulated forms of the MST.

Read More