skip to main content skip to footer

Exploring Methods for Developing Behaviorally Anchored Rating Scales for Evaluating Structured Interview Performance

Author(s):
Kell, Harrison; Martin-Raugh, Michelle; Carney, Lauren; Inglese, Patricia A.; Chen, Lei; Feng, Gary
Publication Year:
2017
Report Number:
RR-17-28
Source:
ETS Research Report
Document Type:
Report
Page Count:
28
Subject/Key Words:
Anchored Scaling, Crowdsourcing, Employment Interviews, Performance Appraisal, Social Skills, Structured Interview (SI), Psychological Assessment, Behavior Rating Scales

Abstract

Behaviorally anchored rating scales (BARS) are an essential component of structured interviews. Use of BARS to evaluate interviewees’ performance is associated with greater predictive validity and reliability and less bias. BARS are time-consuming and expensive to construct, however. This report explores the feasibility of gathering participants’ responses to structured interview questions through an online crowdsourcing platform and using those responses to develop BARS. We describe the development of 12 structured interview questions to assess four applied social skills, elicitation of responses to these questions in the form of critical incidents from 68 respondents, and the creation of BARS from these critical incidents. Results indicate online participants are able to produce responses of sufficient quality to generate BARS for evaluating structured interview performance. We conclude by discussing limitations to this approach and future directions for research and practice.

Read More