skip to main content skip to footer

An Empirical Comparison of Judgmental Approaches to Standard Setting Procedures

Davis, E. Leta; Rock, Donald A.; Werts, Charles E.
Publication Year:
Report Number:
ETS Research Report
Document Type:
Page Count:
Subject/Key Words:
Content Analysis, Decision Making, Qualification Tests, Real Estate Licensing Examination, Scoring, Standards


An experiment was designed that varied cutting score procedures, instructions, and types of judges in order to address the following questions concerning the Real Estate Licensing Examination: (1) will the cutting score levels produced by groups of judges from differing backgrounds (academicians vs. practitioners vs. lawyers) using the same method and instructions be different? (2) will the agreement between item rating profiles vary across these different groups of judges? and (3) does either agreement across items and/or levels vary systematically by instruction/method? It was found that three out of four groups of judges arrived at significantly higher cutting score levels using the Angoff Method than when using the Nedelsky procedure. The Angoff procedure was more effective in setting standards that distinguished the minimally qualified practitioner from the individual with average qualifications. Although the Angoff Method demonstrated somewhat higher interjudge agreement with respect to the patterns of item responses, the average correlation between item profiles was generally low for both procedures.

Read More