skip to main content skip to footer

An Adaptive Algebra Test: A Testlet-based,Hierarchically-Structured Test With Validity-based Scoring CAT

Wainer, Howard; Lewis, Charles; Kaplan, Bruce A.; Braswell, James S.
Publication Year:
Report Number:
ETS Research Report
Document Type:
Page Count:
Subject/Key Words:
Adaptive Testing, Algebra Tests, Computer Assisted Testing, Test Validity


An earlier report (Wainer & Lewis, 1990) described the initial development of a testlet-based algebra test. In this account the details of this excursion into the use of hierarchial testlets and validity-based scoring are provided. A pretest of two 15-item hierarchial testlets was carried out in which examinees' performance on a four-item subset of each testlet was used to predict performance on the entire testlet. Four models for constructing hierarchies were considered. These presentation hierarchies were compared with one another and with an optimally chosen set of four linearly administered items. The comparison was carried out using both the root mean square error and the conditional posterior variance as the criterion. It was found on cross validation that although an adaptive test is everywhere superior to a fixed format test, this superiority is crucially dependent upon the quality of the items. When items vary considerably in quality, a fixed format test, which uses the best items, can do almost as well as an adaptive test of equal length. (29pp.)

Read More