skip to main content skip to footer

Formative Studies of Praxis III: Classroom Performance Assessments- Comparisons of Assessor Approaches to the Observing, Note Taking,Coding, and Selecting Processes (Formative Study B4)

Author(s):
Jones, Jacqueline; Morris, Lori V.
Publication Year:
1994
Report Number:
RM-94-04
Source:
ETS Research Memorandum
Document Type:
Report
Page Count:
80
Subject/Key Words:
Classroom Observation Techniques, Interrater Reliability, Performance Assessment, Praxis Series, Program Evaluation, Teacher Evaluation

Abstract

The developers planned to use the results of the research to guide them in making informed changes in the criteria descriptions, the scoring rules, the assessment instruments, the procedural guidelines, and the assessor training program. In this study, assessors evaluated their own assessment documentation (i.e., their classroom observation notes, interview notes, and Record-of- Evidence form). They completed a work sheet that guided them through the evaluation of those documents and provided feedback on the extent to which they had met some of the goals of assessor training. The work sheets were collected, and the assessors' responses were analyzed in order to identify and examine differences across assessors in the perceived quality of their documentation. Assessors captured much teacher talk, teacher action/behavior, and teacher/student interaction in their classroom observation notes, but they did not capture as much student talk, student action/behavior, and student/student interaction, or provide much narrative description. Assessors noted a number of problems that arose when taking notes during classroom observations (e.g., not hearing all that was taking place, having difficulty observing and writing at the same time) and conducting interviews (e.g., having difficulty eliciting enough information from the candidate, remaining attentive to the candidate while taking notes). Overall, assessors felt that their classroom observation notes and interview notes provided moderately detailed and comprehensive coverage of what occurred in the classroom and during interviews. A number of assessors expressed some reservation about coming back to their notes in a year to reconstruct what happened in the observation, particularly assessors at the third site. Assessors used only one documentation source to provide evidence for certain criteria. In a number of cases, assessors had evidence from multiple documentation sources for a given criterion, but when they wrote up the Record of Evidence, they included evidence form only one source because that source provided the most compelling evidence in support of the rating. For most criteria, most assessors included examples of specific behaviors, events, interactions, etc. and provided quotes, but some assessors had problems citing direct evidence for a few criteria.

Read More