skip to main content skip to footer

Automated Tools for Subject Matter Expert Evaluation of Automated Scoring

Author(s):
Williamson, David M.; Bejar, Isaac I.; Sax, Anne
Publication Year:
2004
Report Number:
RR-04-14
Source:
ETS Research Report
Document Type:
Report
Page Count:
42
Subject/Key Words:
Automated Scoring, Computerized Simulations, Classification, Regression, Neural Networks, Human Scoring, Human Computer Agreement, Quality Control

Abstract

Three applications comprise this investigation, the first of which suggests that CART can facilitate efficient and economical identification of specific elements of complex solutions that contribute to automated and human score discrepancies. The second application builds on the first by exploring CART?s value for efficiently and accurately automating case selection for human intervention to ensure score validity. The final application explores the potential for SOM to reduce the need for SMEs in evaluating automated scoring. While both supervised and unsupervised methodologies examined were found to be promising tools for facilitating SME roles in maintaining and improving the quality of automated scoring, such applications remain unproven and further studies are necessary to establish the reliability of these techniques.

Read More