angle-up angle-right angle-down angle-left close user menu open menu closed search globe bars phone store

Supporting Efficient, Evidence-Centered Item Development for the GRE Verbal Measure

Sheehan, Kathleen M.; Kostin, Irene W.; Futagi, Yoko
Publication Year:
Report Number:
GRE Board Research Report 03-14, RR-07-29
ETS Research Report
Document Type:
Page Count:
Subject/Key Words:
Automated Scoring and Natural Language Processing Evidence Centered Design (ECD) Graduate Record Examination (GRE) Item Modeling Test Items Verbal Ability


This paper explores alternative approaches for facilitating efficient, evidence-centered item development for a new type of verbal reasoning item developed for use on the GRE General Test. Results obtained in two separate studies are reported. The first study documented the development and validation of a fully automated approach for locating the types of stimulus materials needed to support more efficient generation of the studied item type. The validation results confirmed that the proposed filtering technique can help item writers increase the percentage of acceptable stimulus paragraphs located per unit time interval from the current level of about 10% to nearly 30%. The second study documented the development and validation of a set of task models designed to help item writers generate new items that are optimally constructed to provide high-quality evidence about targeted skills. The proposed models were validated by considering the percentage of difficulty variance accounted for by the specified item classifications. That amount ranged from slightly more than 30% for items designed to test vocabulary skills to slightly more than 40% for items designed to test additional verbal reasoning skills, such as generating near and far inferences and understanding complex oppositional reasoning.

Read More