skip to main content skip to footer

Impact of Per-Item Sample Size on Group-Score Proficiency Estimates IRT NAEP SD ELL EM BIB 3PL 2PL

Author(s):
Xi, Nuo; Mazzeo, John
Publication Year:
2023
Report Number:
RM-23-03
Source:
ETS Research Memorandum
Document Type:
Report
Page Count:
41
Subject/Key Words:
Sample Size, Proficiency, Group-Score Assessments, Item Response Theory (IRT), Item Parameters, IRT Item Parameter Estimation, Latent Regression, Calibration, National Assessment of Educational Progress (NAEP), Simulation Methods, Matrix Sampling, Students with Disabilities, English Language Learners (ELL), Ethnicity, Race, Expectation-Maximization Algorithms, Uncertainty, 2-Parameter Logistic (2PL) Model, 3-Parameter Logistic Model (3PL), Balanced Incomplete Block (BIB)

Abstract

The National Assessment of Educational Progress (NAEP) employs an item response theory (IRT)–based latent regression estimation procedure to recover group-score proficiency estimates. At this latent regression stage, item parameter estimates out of the IRT calibration stage are needed to compute the likelihood function, and the current practice is to take the item parameter estimates as fixed and known without considering the uncertainty associated with these estimates. To evaluate if and to what extent this practice introduces additional uncertainty in the estimation process, we conducted a series of simulation studies to calculate the bias, standard error, and root mean square error associated with the final group-score estimates. Results suggest that the amount of additional variation unaccounted for by the current practice is not very large compared to the estimates based on the true item parameter values. However, the estimation process is associated with some difficulty recovering the lower end of the proficiency distribution, especially when the sample size is small. A better understanding of how to accurately recover this portion of the distribution would be an important research direction for the program to follow.

Read More