skip to main content skip to footer

Monitoring Faculty Consultant Performance in the Advanced Placement English Literature and Composition Program with a Many-Faceted Rasch Model IRT AP

Author(s):
Engelhard, George Jr.; Myford, Carol M.
Publication Year:
2003
Report Number:
RR-03-01, CBR-2003-01
Source:
ETS Research Report
Document Type:
Report
Page Count:
60
Subject/Key Words:
Raters, Rater Effects, Performance Assessment, Item Response Theory (IRT), Rasch Measurement, FACETS, Rater Monitoring, Quality Control, Advanced Placement Program (AP), English Literature and Composition

Abstract

The purpose of this study was to examine, describe, evaluate, and compare the rating behavior of faculty consultants who scored essays written for the Advanced Placement English Literature and Composition (AP ELC) exam. Data from the 1999 AP ELC exam were analyzed using FACETS and SAS. The faculty consultants were not all interchangeable in terms of the level of severity they exercised. If students' ratings had been adjusted for severity differences, the AP grades of about 30 percent of the students would have been different from the one they received. Almost all the differences were one grade or less. Adjusting ratings for faculty consultant severity differences would not impact some student subgroups more than others.

Read More