skip to main content skip to footer

Anchored Graphical Representations: A Graphical Alternative to Traditional Just Qualified Candidate Descriptors for Licensure Tests BARS

Author(s):
Kannan, Priya; Tannenbaum, Richard J.; Hebert, Delano
Publication Year:
2018
Report Number:
RR-18-40
Source:
ETS Research Report
Document Type:
Report
Page Count:
19
Subject/Key Words:
Standard Setting, Performance Level Descriptors, Teacher Licensure, Cut Scores, Graphical Representations, Behaviorally Anchored Rating Scales (BARS)

Abstract

A well‐constructed just qualified candidate (JQC) description is needed to arrive at a reasonable passing score for licensure tests. Traditionally, such descriptions consist of a list of knowledge and skill statements without sufficient context to internalize its intended meaning, allowing the standard‐setting panelists to make idiosyncratic interpretations. In a series of studies, we evaluated an alternative JQC description called anchored graphical representation (AGR). AGRs are intended to provide both text and visual organizers by each tested domain to contextualize the meaning of the JQC. In Study 1, 22 mathematics educators participated in a mock standard‐setting study and were randomly assigned to conditions in which they used either the AGR‐based or the text‐based definitions in making standard‐setting judgments. In Study 2, 17 social studies educators were randomly assigned to either the AGR‐based or text‐based condition. While the overall passing score did not noticeably vary between the conditions, consistent with our expectations, results from both studies showed that the domain passing scores for the AGR groups reflected the relative importance of each test domain for the JQC and that the inter‐panelist variability was smaller for the AGR groups. Collectively, our results indicate that use of the AGR resulted in standard‐setting ratings that were more aligned with the panelists' differential expectations of JQCs' performance for each domain, and the results add to the procedural validity evidence supporting the reasonableness of standard‐setting recommendations.

Read More