skip to main content skip to footer

Comparing the Incomparable: An Essay on the Importance of Big Assumptions and Scant Evidence

Author(s):
Wainer, Howard
Publication Year:
1999
Report Number:
RR-99-25
Source:
ETS Research Report
Document Type:
Report
Page Count:
24
Subject/Key Words:
Assumptions (Testing), Comparative Analysis, Differences, Equated Scores

Abstract

Comparisons between groups usually involve making statistical adjustments so that the comparisons are, statistically speaking, ceteris paribus. Usually included within those adjustments are some untestable assumptions. These untestable assumptions are often determined by the goals of the testing program; they are sometimes justified on the basis of theory, sometimes on the basis of political expediency, and sometimes on just a hope and a prayer. If we are vigilant, bits of evidence may emerge from the data that can corroborate or refute our assumptions. In this essay I describe several examples, drawn from mental testing, that illustrate these issues, and suggest a plausible path of investigation that may provide a better estimate of the practical limitations of accuracy for any proposed solution to this vexing equity issue.

Read More