skip to main content skip to footer

Implementing Shaffer's Multiple Comparison Procedure For Large Numbers of Groups NAEP

Author(s):
Donoghue, John R.
Publication Year:
1998
Report Number:
RR-98-47
Source:
ETS Research Report
Document Type:
Report
Page Count:
38
Subject/Key Words:
National Assessment of Educational Progress (NAEP), Shaffer's Procedure, Paired Comparisons, Comparative Analysis

Abstract

Shaffer (1986) presented a multiple comparison procedure for paired comparisons that had superior power to many available alternatives. Unfortunately, use of the procedure has been severely limited by the complexity of its implementation. This paper presents a method of allowing Shaffer's procedure to be used in a much larger class of problems. Theoretical results concerning an aspect of Shaffer's procedure are derived. The use of these results in implementing the procedure is then described. Next, two heuristics are described which greatly enhance the efficiency of the method. Finally, an efficient algorithmic method of implementing Shaffer's procedure is outlined. The present work allows Shaffer's procedure for multiple comparisons to be applied to large numbers of groups. Unlike earlier work, no assumptions about the testing situation need be made. These methods are illustrated by application to two data sets; comparisons of the order of 11 clustering methods, and comparison of the means of 44 states from the 1994 National Assessment of Educational Progress Trial State Assessment of Reading at Grade 4.

Read More