Can There Be Reliability Without "Reliability"?

Author(s):

Mislevy, Robert J.

Publication Year:

1994

Report Number:

RM-94-18-ONR

Source:

ETS Research Memorandum

Document Type:

Report

Page Count:

24

Subject/Key Words:

Office of Naval Research, Educational Assessment, Test Reliability, Test Validity

Abstract

A recent article by Pamela Moss asks the title question, "Can there be validity without reliability?" If by reliability we mean only KR-20 coefficients or inter-rater correlations, the answer is yes. Sometimes these particular indices for evaluating evidence suit the problem we encounter; sometimes they don't. If by reliability we mean credibility of evidence, where credibility is defined as appropriate to the intended inference, the answer is no, we cannot have validity without reliability. Because "validity" encompasses the process of reasoning as well as the data, uncritically accepting observations as strong evidence, when they may be incorrect, misleading, unrepresentative, or fraudulent, may lead coincidentally to correct conclusions but not to valid ones. This paper discusses and illustrates a broader conception of "reliability" in educational assessment, to ground a deeper understanding of the issues raised by Professor Moss's question.

Read More

• Request Copy (specify title and report number, if any)