skip to main content skip to footer

Cognitive Processing Requirements of Constructed Figural Response and Multiple-Choice Items in Architecture Assessment NCARB

Martinez, Michael E.; Katz, Irvin R.
Publication Year:
Report Number:
ETS Research Report
Document Type:
Page Count:
Subject/Key Words:
National Council of Architectural Registration Boards (NCARB), Architecture Tests, Cognitive Processes, Constructed Responses, Figural Response Items, Graphs, Test Format


Contrasts between constructed response items and stem-equivalent multiple-choice counterparts have yielded only a few weak generalizations. Such comparisons typically have involved averaging item characteristics, and this aggregation has masked differences in statistical properties at the item level. Moreover, even aggregated format differences have not been explained in terms of differential cognitive processing demands of the items. This paper examines item-level differences between figural response items and their multiple-choice counterparts in the domain of architecture. The figural response item format is an assessment form that uses figural material (such as graphs, illustrations, and diagrams) as item stimuli and the medium through which knowledge and skill are demonstrated. We first examine item-level format differences in difficulty and then ask whether there are corresponding differences in the cognitive processing requirements of the items that can account for the psychometric differences. After finding evidence for these connections, we propose that differences in processing requirements and concomitant psychometric properties might be systematic and predictable. These analyses are important in a larger sense in that they shed light on aspects of construct validity that are frequently neglected and they touch the interface of the usually segregated psychometric and cognitive methodologies. (28pp.)

Read More