skip to main content skip to footer

Modeling Writing Traits in a Formative Essay Corpus NLP AES

Author(s):
Deane, Paul; Yan, Duanli; Castellano, Katherine E.; Attali, Yigal; LaMar, Michelle M.; Zhang, Mo; Blood, Ian; Bruno, James V.; Li, Chen; Cui, Wenju; Ruan, Chunyi; Appel, Colleen; James, Kofi; Long, Rodolfo; Qureshi, Farah
Publication Year:
2024
Report Number:
RR-24-02
Source:
ETS Research Report
Document Type:
Report
Page Count:
64
Subject/Key Words:
Writing Assessment, Writing Tasks, Writing Processes, Natural Language Processing (NLP), Automated Essay Scoring (AES), Automated Scoring of Writing Quality, Factor Analysis, Structural Equation Modeling, Multidimensional Analysis, Multidimensional Tests, Corpus Linguistics, Genre, Confirmatory Factor Analysis, Criterion Online Writing Evaluation, e-rater, Elementary Secondary Education

Abstract

This paper presents a multidimensional model of variation in writing quality, register, and genre in student essays, trained and tested via confirmatory factor analysis of 1.37 million essay submissions to ETS’ digital writing service, Criterion. The model was also validated with several other corpora, which indicated that it provides a reasonable fit for essay data from 4th grade to college. It includes an analysis of the test-retest reliability of each trait, longitudinal trends by trait, both within the school year and from 4th to 12th grades, and analysis of genre differences by trait, using prompts from the Criterion topic library aligned with the major modes of writing (exposition, argumentation, narrative, description, process, comparison and contrast, and cause and effect). It demonstrates that many of the traits are about as reliable as overall e-rater scores, that the trait model can be used to build models somewhat more closely aligned with human scores than standard e-rater models, and that there are large, significant trait differences by genre, consistent with genre differences in trait patterns described in the larger literature. Some of the traits demonstrated clear trends between successive revisions. Students using Criterion appear to have consistently improved grammar, usage, and spelling after getting Criterion feedback and to have marginally improved essay organization. Many of the traits also demonstrated clear grade level trends. These features indicate that the trait model could be used to support more detailed scoring and reporting for writing assessments and learning tools.

Read More