skip to main content skip to footer

The Influence of Changes in Assessment Design on the Psychometric Quality of Scores NBPTS

Author(s):
Wolfe, Edward W.; Gitomer, Drew H.
Publication Year:
2000
Report Number:
RM-00-01
Source:
ETS Research Memorandum
Document Type:
Report
Page Count:
35
Subject/Key Words:
National Board for Professional Teaching Standards (NBPTS), Performance Assessment, Test Construction, Interrater Reliability, Scores, Teachers, Reliability

Abstract

This article addresses the problem of improving the measurement quality of a complex performance assessment through principled assessment design. We describe the characteristics and measurement impact of steps taken to improve assessment exercise design along with modifications in assessor training materials and procedures between the 1995-1996 and the 1996-1997 administrations of the National Board for Professional Teaching Standards Early Childhood/Generalist examination. Specifically, we describe how the revision of this assessment resulted in increases in the inter- assessor agreement, internal consistency, and generalizability of scores. All indices we examined improved as a result of the revisions. The results suggest that previously observed limits on the measurement quality of performance assessments due to the relatively small number of items that contribute to an assessment score can be altered significantly through attention to assessment design and related scoring processes.

Read More