skip to main content skip to footer

Evaluation of Different Scoring Rules for a Noncognitive Test in Development IRT

Author(s):
Guo, Hongwen; Zu, Jiyun; Kyllonen, Patrick C.; Schmitt, Neal
Publication Year:
2016
Report Number:
RR-16-03
Source:
ETS Research Report
Document Type:
Report
Page Count:
13
Subject/Key Words:
Scoring, Item Response Curve (IRC), Item Response Theory (IRT), Test Reliability, Noncognitive Assessment, Situational Judgment Tests (SJT)

Abstract

In this report, systematic applications of statistical and psychometric methods are used to develop and evaluate scoring rules in terms of test reliability. Data collected from a situational judgment test are used to facilitate the comparison. For a well-developed item with appropriate keys (i.e., the correct answers), agreement among various item-scoring rules is expected in the item-option characteristic curves. In addition, when models based on item-response theory fit the data, test reliability is greatly improved, particularly if the nominal response model and its estimates are used in scoring.

Read More