Potential Impact of Context Effects on the Scoring and Equating of the Multistage GRE® Revised General Test

Author(s):
Davey, Tim; Lee, Yi-Hsuan
Publication Year:
2011
Report Number:
RR-11-26, GREB-08-01
Source:
Document Type:
Subject/Key Words:
context effects, position effects, randomization

Abstract

Both theoretical and practical considerations have led the revision of the Graduate Record Examinations® (GRE®) revised General Test, here called the rGRE, to adopt a multistage adaptive design that will be continuously or nearly continuously administered and that can provide immediate score reporting. These circumstances sharply constrain the available options for equating or linking together the thousands of unique test forms that will need to be administered each year. The only practical method is to pre-equate each form through item response theory (IRT) methods. Unfortunately, doing so means that item position and context effects are both vitally important and potentially difficult to control. This study examines item position effects in a GRE context and attempts to assess the extent to which they may affect the rGRE. It was found that position effects are in fact present in GRE data and that they can be particularly troublesome under multistage testing. Pretest strategies are able to mitigate effects to some degree, but complete control will require careful attention be paid to the time limits and associated test configuration details in the design of the rGRE.

Read More