An Investigation of the Impact of Misrouting Under Two-Stage Multistage Testing: A Simulation Study

Author(s):
Kim, Sooyeon; Moses, Tim P.
Publication Year:
2014
Report Number:
RR-14-01
Source:
ETS Research Report
Document Type:
Report
Page Count:
13
Subject/Key Words:
Adaptive Tests Multistage Testing (MST) Panel Assembly Routing Simulation Studies Test Bias

Abstract

The purpose of this study was to investigate the potential impact of misrouting under a 2-stage multistage test (MST) design, which includes 1 routing and 3 second-stage modules. Simulations were used to create a situation in which a large group of examinees took each of the 3 possible MST paths (high, middle, and low). We compared differences in examinees' scores associated with different paths under 2 MST assembly conditions: a small-difference condition in which 3 modules at Stage 2 overlap in difficulty, and a large-difference condition in which the 3 modules at Stage 2 are distinct in difficulty. We also compared examinees' score differences associated with the target MST path with the ones obtained when the second-stage module was 1 level of difficulty above or below the target module. When the second-stage modules overlapped in difficulty, the score differences (i.e., bias) associated with different MST paths were negligible for practical purposes. Similar trends appeared even when the second-stage modules were significantly distinct in difficulty. The impact of misrouting was generally minimal under the MST design used in this study.

Read More