skip to main content skip to footer

Formative Studies of Praxis III: Classroom Performance Assessments- Not all Assessors Are Alike: A Case Study of Two Assessors (Formative Study B2)

Author(s):
Reynolds, Anne
Publication Year:
1994
Report Number:
RM-94-02
Source:
ETS Research Memorandum
Document Type:
Report
Page Count:
55
Subject/Key Words:
Classroom Observation Techniques, Interrater Reliability, Performance Assessment, Praxis Series, Program Evaluation, Teacher Evaluation

Abstract

The developers planned to use the results of the research to guide them in making informed changes in the criteria descriptions, the scoring rules, the assessment instruments, the procedural guidelines, and the assessor training program. The study described in this report involved an in- depth look at how two assessors used the measurement tools to make their judgments. The goals of the study were: to understand how assessors execute the tasks of taking and coding notes, selecting evidence from notes to include on the Record-of-Evidence form, and analyzing and weighing that evidence to arrive at a final rating for a criterion; to develop some hypotheses about important similarities and differences in how assessors process information when using a high- inference assessment system; to understand how differences in approach can affect the ratings assessors give on four criteria, one from each domain; and to provide recommendations for improving the assessment process, especially the training of assessors. Five important similarities and differences in how assessors process information when using a high- inference assessment system and how differences in approach can affect the ratings assessors give emerged from the study: (a) Assessors differ in the ways they collect evidence during the observation; (b) assessors differ in the types of interpretations they make when they code their notes; (c) assessors differ in the extent to which they use their coded notes on the Record-of-Evidence form; (d) assessors differ according to the extent to which they have and use special understanding of students, subject matter, school context, etc.; and (e) assessors may differ according to the understandings they have of the rating scale. Recommendations are offered to improve the assessment process, especially the training of assessors.

Read More