skip to main content skip to footer

Can There Be Reliability Without "Reliability"?

Mislevy, Robert J.
Publication Year:
Report Number:
ETS Research Memorandum
Document Type:
Page Count:
Subject/Key Words:
Office of Naval Research, Educational Assessment, Test Reliability, Test Validity


A recent article by Pamela Moss asks the title question, "Can there be validity without reliability?" If by reliability we mean only KR-20 coefficients or inter-rater correlations, the answer is yes. Sometimes these particular indices for evaluating evidence suit the problem we encounter; sometimes they don't. If by reliability we mean credibility of evidence, where credibility is defined as appropriate to the intended inference, the answer is no, we cannot have validity without reliability. Because "validity" encompasses the process of reasoning as well as the data, uncritically accepting observations as strong evidence, when they may be incorrect, misleading, unrepresentative, or fraudulent, may lead coincidentally to correct conclusions but not to valid ones. This paper discusses and illustrates a broader conception of "reliability" in educational assessment, to ground a deeper understanding of the issues raised by Professor Moss's question.

Read More