skip to main content skip to footer

An Investigation of the Validity of an Empirical Bayes Approach to Mantel-Haenszel DIF Analysis DIF QLI-MH

Zwick, Rebecca J.; Thayer, Dorothy T.; Lewis, Charles
Publication Year:
Report Number:
ETS Research Report
Document Type:
Page Count:
Subject/Key Words:
Bayesian Statistics, Differential Item Functioning (DIF), Mantel-Haenszel Technique, Statistical Analysis, Validity


We investigated the utility of an empirical Bayes (EB) enhancement to Mantel-Haenszel (MH) differential item functioning (DIF) analysis. We assume that the MH statistics are normally distributed and that the prior distribution of underlying DIF parameters is also normal. We use the posterior distribution of DIF parameters to make inferences about the item's true DIF status (the true DIF method) and the posterior predictive distribution to predict the item's future observed status (the future DIF method). DIF status is expressed in terms of the probabilities associated with each of the five DIF levels defined by the ETS classification system: C-, B-, A, B+, and C+. The proposed methods have several possible advantages. First, the use of prior information allows for more stable DIF estimation, especially in small samples-a feature that should be advantageous in computer-adaptive testing. The EB approach may convey information about DIF stability in a more useful way. The results represent the state of knowledge about an item's true DIF status as probabilistic, not deterministic. The EB DIF results lend themselves well to graphic display, which may be useful in conveying findings to test developers and DIF committees. Also, results can be projected for future administration conditions that may differ from the present ones. As an alternative method of DIF flagging, we also investigated the use of a loss function to identify potential DIF items; this approach is very general and is not linked to the ETS DIF classification system. A possible advantage of the loss function procedure over the ETS classification system is its greater flexibility.

Read More