skip to main content skip to footer

Computerized Mastery Testing With Nonequivalent Testlets IRT NCARB

Author(s):
Sheehan, Kathleen M.; Lewis, Charles
Publication Year:
1990
Report Number:
RR-90-16
Source:
ETS Research Report
Document Type:
Report
Page Count:
44
Subject/Key Words:
Bayesian Statistics, Computer Assisted Testing, Decision Theory, Item Response Theory (IRT), Mastery Tests, National Council of Architectural Registration Boards (NCARB)

Abstract

In this report, practical procedure for determining the effect of testlet nonequivalence on the operating characteristics of a testlet-based computerized mastery test (CMT) is introduced. The procedure involves estimating the CMT decision rule twice, once with testlets treated as equivalent and once with testlets treated as nonequivalent. In the equivalent testlet mode, the likelihood functions estimated for specific number right scores are assumed to be constant across testlets, and a single set of cut scores is estimated for all testlets. In the nonequivalent testlet mode, the likelihood functions estimated for specific number-right scores are allowed to vary from one testlet to another, and a different set of cut scores is estimated for each permutation of testlet presentation order. Small differences between the estimated operating characteristics of the equivalent testlet decision rule and the nonequivalent testlet decision rule indicate that the assumption of equivalent testlets was warranted. This procedure is illustrated with data from the Architect Registration Examination, a professional certification examination administered by the National Council of Architectural Registration Boards (NCARB). (48pp.)

Read More