skip to main content skip to footer

Toward the Automated Scoring of Written Arguments: Developing an Innovative Approach for Annotation

Author(s):
Song, Yi; Deane, Paul; Beigman Klebanov, Beata
Publication Year:
2017
Report Number:
RR-17-11
Source:
ETS Research Report
Document Type:
Report
Page Count:
17
Subject/Key Words:
Automated Scoring and Natural Language Processing, Annotation, Critical Thinking, Argumentation Schemes, Higher Education, Workplace Skills, Automated Scoring of Written Content, Writing Assessment, annotator training

Abstract

This project focuses on laying the foundations for automated analysis of argumentation schemes, supporting identification and classification of the arguments being made in a text, for the purpose of scoring the quality of written analyses of arguments. We developed annotation protocols for 20 argument prompts from a college-level test under the framework of the theory of argumentation schemes, which defines reasoning patterns in argumentation. The annotation protocols listed critical questions associated with each argumentation scheme, which makes the argument structure in a text explicit and classifiable. Furthermore, we annotated 200 student essays across four selected argument prompts to test if the annotation protocols can be applied reliably by human annotators. Preliminary results indicate that this method of analyzing argument structure is reliable and promising.

Read More