angle-up angle-right angle-down angle-left close user menu open menu closed search globe bars phone store

Evaluation of Different Scoring Rules for a Noncognitive Test in Development

Author(s):
Guo, Hongwen; Zu, Jiyun; Kyllonen, Patrick C.; Schmitt, Neal
Publication Year:
2016
Report Number:
RR-16-03
Source:
ETS Research Report
Document Type:
Report
Page Count:
13
Subject/Key Words:
Item Response Curve (IRC) Item Response Theory (IRT) Noncognitive Assessment Scoring Situational Judgment Tests (SJT) Test Reliability

Abstract

In this report, systematic applications of statistical and psychometric methods are used to develop and evaluate scoring rules in terms of test reliability. Data collected from a situational judgment test are used to facilitate the comparison. For a well-developed item with appropriate keys (i.e., the correct answers), agreement among various item-scoring rules is expected in the item-option characteristic curves. In addition, when models based on item-response theory fit the data, test reliability is greatly improved, particularly if the nominal response model and its estimates are used in scoring.

Read More