skip to main content skip to footer

Extending the Rule Space Model to a Semantically-Rich Domain: Diagnostic Assessment in Architecture

Author(s):
Katz, Irvin R.; Martinez, Michael E.; Sheehan, Kathleen M.; Tatsuoka, Kikumi K.
Publication Year:
1993
Report Number:
RR-93-42-ONR
Source:
ETS Research Report
Document Type:
Report
Page Count:
32
Subject/Key Words:
Architecture Tests, Classification, Computer Assisted Testing, Educational Diagnosis, Item Characteristics, Models, Problem Solving, Rule-Space Model, Test Items

Abstract

(38pp.) This paper presents a technique for applying the Rule Space model of cognitive diagnosis (Tatsuoka, 1983) to assessment in a semantically rich domain. Responses to 22 architecture test items, developed to assess a range of architectural knowledge, were analyzed using Rule Space. Verbal protocol analyses guided the construction of a model of examinee performance, consisting of processes for constructing an initial representation of an item (labeled understand), forming goals and performing actions based on those goals (solve), and determining whether goals have been attempted and satisfied (check). Item attributes, derived from these processes, formed the basis for diagnosis. Our technique extends Rule Space's applicability by defining attributes in terms of item characteristics and the causal relations between characteristics and the problem-solving model. Data were collected from 122 architects of various ability levels (students, architecture interns, and professional architects). Rule Space successfully classified approximately 65%, 90%, and 40% of examinees based, respectively, on attributes associated with the understand, solve, and check processes of the problem- solving model. The findings support the effectiveness of Rule Space in a complex domain and suggest directions for developing new architecture items by using attributes particularly effective at distinguishing among examinees of different ability levels.

Read More