skip to main content skip to footer

Unfair Treatment vs. Confirmation Bias? Comments on Santelices and Wilson DIF SEA

Author(s):
Dorans, Neil J.
Publication Year:
2010
Report Number:
RR-10-20
Source:
ETS Research Report
Document Type:
Report
Page Count:
19
Subject/Key Words:
Confirmation Bias, Differential Item Functioning (DIF), Differential Prediction, Score Equity Assessment (SEA), Test Fairness

Abstract

Santelices and Wilson (2010) claimed to have addressed technical criticisms of Freedle (2003) presented in Dorans (2004a) and elsewhere. Santelices and Wilson’s abstract claimed that their study confirmed that SAT® verbal items do function differently for African American and White subgroups. In this commentary, I demonstrate that the Santelices and Wilson article provided no evidence to confirm differential item functioning (DIF) and failed to address my technical criticisms of Freedle. Instead, Santelices and Wilson contained several misrepresentations, including substituting “considered serious” for “more unusual” to describe an effect size and claiming to have studied four editions of the SAT when only two were actually studied. Central to its thesis was a hypothesis about DIF/difficulty correlations that was misattributed to Dorans. Santelices and Wilson based their argument for DIF on correlations between highly correlated variations on an index of DIF with item difficulty. While failing to demonstrate either unfairness in the test items or unfairness in the treatment of Freedle, Santelices and Wilson did contain evidence of confirmation bias.

Read More