Supporting Efficient, Evidence-Centered Item Development for the GRE® Verbal Measure

Author(s):
Sheehan, Kathleen M.; Kostin, Irene; Futagi, Yoko
Publication Year:
2007
Report Number:
RR-07-29, GREB-03-14
Source:
Document Type:
Subject/Key Words:
Evidence-centered design ECD item difficulty modeling latent semantic analysis LSA targeted item writing tree-based regression the SourceFinder system verbal reasoning skills vocabulary knowledge

Abstract

This paper explores alternative approaches for facilitating efficient, evidence-centered item development for a new type of verbal reasoning item developed for use on the GRE® General Test. Results obtained in two separate studies are reported. The first study documented the development and validation of a fully automated approach for locating the types of stimulus materials needed to support more efficient generation of the studied item type. The validation results confirmed that the proposed filtering technique can help item writers increase the percentage of acceptable stimulus paragraphs located per unit time interval from the current level of about 10% to nearly 30%. The second study documented the development and validation of a set of task models designed to help item writers generate new items that are optimally constructed to provide high-quality evidence about targeted skills. The proposed models were validated by considering the percentage of difficulty variance accounted for by the specified item classifications. That amount ranged from slightly more than 30% for items designed to test vocabulary skills to slightly more than 40% for items designed to test additional verbal reasoning skills, such as generating near and far inferences and understanding complex oppositional reasoning.

Read More