skip to main content skip to footer

A Differential Word Use Measure for Content Analysis in Automated Essay Scoring

Author(s):
Attali, Yigal
Publication Year:
2011
Report Number:
RR-11-36
Source:
ETS Research Report
Document Type:
Report
Page Count:
19
Subject/Key Words:
Automated Scoring, Content Analysis, Writing Assessment, Electronic Essay Rater (E-rater), Automated Scoring and Natural Language Processing

Abstract

This paper proposes an alternative content measure for essay scoring, based on the difference in the relative frequency of a word in high-scored versus low-scored essays. The differential word use (DWU) measure is the average of these differences across all words in the essay. A positive value indicates the essay is using vocabulary more typical of high-scoring essays in this task, and vice versa. In addition to the traditional prompt level, this measure is also computed at the generic task level, where the content of an essay is evaluated in the context of the general vocabulary of the writing task (e.g., GRE issue), across different prompts. Evaluation results across four GRE and TOEFL tasks are presented. Factor analyses show that both the prompt and task DWU measures load on the same factor as the prompt-specific content analysis measures of e-rater. Regression results on the human essay scores show that the generic task DWU measure is a strong predictor of human scores, second only to essay length among noncontent e-rater features. This measure provides a way to conduct a task-level content analysis that is related to the prompt-specific content analysis of e-rater but does not require prompt-specific training. Regression results for the prompt-level DWU show that it can be used as a replacement to prompt-specific content-vector analysis (CVA) features.

Read More