skip to main content skip to footer

Implementing a Contributory Scoring Approach for the GRE Analytical Writing Section: A Comprehensive Empirical Investigation GRE AES

Author(s):
Breyer, F. Jay; Rupp, Andre A.; Bridgeman, Brent
Publication Year:
2017
Report Number:
RR-17-14
Source:
ETS Research Report
Document Type:
Report
Page Count:
30
Subject/Key Words:
Automated Scoring and Natural Language Processing, Check Scoring, Contributory Scoring, Graduate Record Examination (GRE), Analytical Writing, Writing Assessment, Empirical Investigations, Linear Regression, ScoreSelect, Assessment Design, Human Scoring, Automated Essay Scoring (AES), Rater Reliability, e-rater

Abstract

In this research report, we present an empirical argument for the use of a contributory scoring approach for the 2-essay writing assessment of the analytical writing section of the GRE test in which human and machine scores are combined for score creation at the task and section levels. The approach was designed to replace a currently operational all-human check scoring approach in which machine scores are used solely as quality-control checks to determine when additional human ratings are needed due to unacceptably large score discrepancies. We use data from 6 samples of essays collected from test takers during operational administrations and special validity studies to empirically evaluate 6 different score computation methods. During the presentation of our work, we critically discuss key methodological design decisions and underlying rationales for these decisions. We close the report by discussing how the research methodology is generalizable to other testing programs and use contexts.

Read More