skip to main content skip to footer

The Effect of Small Calibration Sample Sizes on TOEFL IRT-Based Equating IRT TOEFL

Carey, Patricia A.; Tang, K. Linda; Way, Walter D.
Publication Year:
Report Number:
ETS Research Report
Document Type:
Page Count:
Subject/Key Words:
BILOG, Equated Scores, Item Response Theory (IRT), LOGIST, Sample Size, Scaling, Test of English as a Foreign Language (TOEFL)


The present study compared the performance of LOGIST and BILOG on TOEFL® IRT-based scaling and equating using both real and simulated data and two calibration structures. Applications of IRT for the TOEFL program are based on the three-parameter logistic (3PL) model. The results of the study show that item parameter estimates obtained from the smaller real data sample sizes were more consistent with the larger sample estimates when based on BILOG than when based on LOGIST. In addition, the root mean squared error statistics suggest that the BILOG estimates for the item parameters and item characteristic curves were closer in magnitude to the "true" parameter values than were the LOGIST estimates. The equating results based on the parameter estimates suggest that the rule of thumb recommendation that pretest sample sizes be at least 1000 for LOGIST should be retained if at all possible.

Read More