skip to main content skip to footer

Toward a Framework for Constructed-Response Items

Bennett, Randy Elliot; Ward, William C.; Rock, Donald A.; Lahart, Colleen M.
Publication Year:
Report Number:
ETS Research Report
Document Type:
Page Count:
Subject/Key Words:
Constructed-Response Tests, Item Types, Test Construction, Test Format


In this study, a framework for categorizing constructed-response items was developed in which items were ordered on a continuum from multiple-choice to presentation/performance according to the degree of constraint placed on the examinee's response. Two investigations were carried out to evaluate the validity of this framework. In the first investigation, 27 test development staff assigned 46 items of various formats to the categories. Overall, agreement with the intended item categorizations was good, with a median of two of a possible 27 judges disagreeing with a given item's classification. In the second investigation, responses of 40 examinees each to four sets of items were scored by test development staff, with each set scored by four individuals. Results showed scoring agreement to be highest for a category requiring the examinee to choose a response from an extended stimulus array and lowest for items requiring that the stimulus be reordered to form a correct sequence. Whether the reported agreement levels represent sufficient accuracy to permit the widespread use of such items in standardized tests depends on whether some degree of scoring error, however small, can be accepted. (65pp.)

Read More