skip to main content skip to footer

Different Methods of Adjusting for Form Difficulty Under the Rasch Model: Impact on Consistency of Assessment Results IRT

Author(s):
Manna, Venessa F.; Gu, Lixiong
Publication Year:
2019
Report Number:
RR-19-08
Source:
ETS Research Report
Document Type:
Report
Page Count:
18
Subject/Key Words:
Rasch Model, Item Response Theory (IRT), Equating, Outliers (Statistics), Calibration, Test Forms, Item Difficulty

Abstract

When using the Rasch model, equating with a nonequivalent groups anchor test design is commonly achieved by adjustment of new form item difficulty using an additive equating constant. Using simulated 5‐year data, this report compares 4 approaches to calculating the equating constants and the subsequent impact on equating results. The 4 approaches are mean difference, mean difference with outlier removal using the 0.3 logit rule, mean difference with robust z statistic, and the information‐weighted mean difference. Factors studied included sample size, anchor test length, percentage of anchor items displaying outlier behavior, and the distribution of test item difficulty relative to examine ability. The results indicated that the mean difference and information‐weighted mean difference methods performed similarly across all conditions. In addition, with larger sample sizes, the mean difference with 0.3 logit method performed similarly to these 2 methods. The mean difference with robust z method performed most differently from the other three methods of calculating the equating constant. This method removed a large percentage of the anchor items compared to the mean difference with 0.3 logit method but seemed to produce the most stable trend in performance classification across the 5 years, particularly when sample sizes were large.

Read More