Automated Scoring of Writing Quality
The e-rater® automated scoring engine is ETS's proven capability for automated scoring of expository, persuasive and summary essays, currently used in multiple high-stakes programs. The e-rater engine is used in combination with human raters to score the Writing sections of the TOEFL® and GRE® tests, as psychometric research has demonstrated that this combination is superior to either machine scoring or human scoring on their own. It is also used as the sole score in lower-stakes contexts, such as formative use in a classroom setting with ETS's Criterion® online essay evaluation system. The e-rater engine is used to generate the individualized feedback provided to users of Criterion. The e-rater engine addresses the need for essay scoring that is reliable, valid, fast and flexible, as more and more testing programs, including large-volume state testing, move to online delivery and adopt essay-based tasks for writing assessment.
ETS has conducted over a decade of ground-breaking research in natural language processing related to the automated identification of text features characteristic of developing writers. The patented e-rater engine — a platform that automatically provides a rich set of underlying linguistic representations related to writing quality, in addition to scores — represents the culmination of this research to date.
The e-rater engine predicts essay scores based on features related to writing quality, including:
- errors in grammar (e.g., subject-verb agreement)
- usage (e.g., preposition selection)
- mechanics (e.g., capitalization)
- style (e.g., repetitious word use)
- discourse structure (e.g., presence of a thesis statement, main points)
- vocabulary usage (e.g., relative sophistication of vocabulary)
The e-rater engine also includes features related to vocabulary, content appropriateness, organization and development. The e-rater engine's score predictions have been shown to correlate strongly with the scores of human raters and other measures of writing ability. It can also automatically detect responses that are off-topic or otherwise anomalous, and therefore should not be scored.
ETS has an active research agenda that investigates new genres (digital writing formats, such as blogs) and the development of linguistic features suitable for modeling aspects of content and argumentation that reflect additional components of writing quality, such as:
- metrics of text coherence
- organization of claims and evidence
- the writer's stance toward the test question
- the identification of particular topics addressed in the response
- the use of supporting facts from external sources
As e-rater research continues, this research agenda aims to grow the array of writing genres that can be addressed; work with new genres and features to support the writing of English learners (e.g., correct use of articles, prepositions and collocations); and advance the state of the art with regard to evaluating the quality of argumentation across different genres and modes of discourse.
Below are some recent or significant publications that our researchers have authored on the subject of automated scoring of writing quality.
Handbook of Automated Essay Evaluation: Current Applications and New Directions
M. D. Shermis & J. Burstein
This comprehensive, interdisciplinary handbook reviews the latest methods and technologies used in automated essay evaluation (AEE) methods and technologies. New York: Routledge. View citation record >
Robust Systems for Preposition Error Correction Using Wikipedia Revisions
A. Cahill, N. Madnani, J. Tetreault, & D. Napolitano
In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 507–517, Atlanta, Ga.
This paper addresses the lack of generalizability in preposition error correction systems across different test sets. The authors then present a large new annotated corpus to be used in training such systems, and illustrate the use of the corpus in training systems across three separate test sets. View citation record >
Detecting Missing Hyphens in Learner Text
A. Cahill, M. Chodorow, S. Wolff and N. Madnani
In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 300–305, Atlanta, Ga.
This paper presents a method for automatically detecting missing hyphens in English text. View citation record >
Automated Scoring of a Summary-Writing Task Designed to Measure Reading Comprehension
N. Madnani, J. Burstein, J. Sabatini, & T. O’Reilly
In Proceedings of the North American Association for Computational Linguistics Eighth Workshop Using Innovative NLP for Building Educational Applications, Atlanta, Ga., June 13, 2013
This paper introduces a cognitive framework for measuring reading comprehension that includes the use of novel summary writing tasks. View citation record >
The e-rater® Automated Essay Scoring System
J. Burstein, J. Tetreault, & N. Madnani
In M. D. Shermis, & J. Burstein (Eds.), Handbook of Automated Essay Scoring: Current Applications and Future Directions. New York: Routledge.
This handbook chapter includes a description of the e-rater automated essay scoring system and its NLP-centered approach, and a discussion of the system’s applications and development efforts for current and future educational settings. View citation record >
A Fast and Flexible Architecture for Very Large Word n-gram Datasets
Natural Language Engineering, FirstView online publication, pp. 1–33
This paper presents a versatile architecture that uses a novel trie-based architecture, features lossless compression, and optimizes both speed and memory use. View citation record >
Correcting Comma Errors in Learner Essays, and Restoring Commas in Newswire Text
R. Israel, J. Tetreault, and M. Chodorow (2012)
Proceedings of the 2012 Meeting of the North American Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pp. 284–294
Association for Computational Linguistics
The authors present a system for detection and correction of the placement of commas in English-language sentences. The system likewise can restore commas in well-crafted sentences. View citation record >
On Using Context for Automatic Correction of Non-Word Misspellings in Student Essays
M. Flor and Y. Futagi
Proceedings of the 7th Workshop on Innovative Use of Natural Language Processing for Building Educational Applications (BEA) pp. 105–115
The authors discuss a new system for spell-checking that uses contextual information to perform automatic correction of non-word misspellings. The article relates how the system has been evaluated against a large body of TOEFL® and GRE® essays, which were written by both native and nonnative English speakers. View citation record >
Using Parse Features for Preposition Selection and Error Detection
J. Tetreault, J. Foster, & M. Chodorow
Proceedings of the 2010 Association for Computational Linguistics (ACL 2010)
Association for Computational Linguistics
This paper evaluates the effect of adding features that aim to improve the detection of preposition errors in writing from speakers of English as a second language. View citation record >
Progress and New Directions in Technology for Automated Essay Evaluation
J. Burstein & M. Chodorow
The Oxford Handbook of Applied Linguistics, 2nd Edition, pp. 487–497
Editor: R. Kaplan
Oxford University Press
This ETS-authored work is part of a 39-chapter volume that covers topics in applied linguistics with the goal of providing a survey of the field, showing the many connections among its subdisciplines, and exploring likely directions of its future development. View citation record >
Using Entity-Based Features to Model Coherence in Student Essays
J. Burstein, J. Tetreault, & S. Andreyev
Human language technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pp. 681–684
Association for Computational Linguistics
This paper describes a study in which researchers combined an algorithm for observing what computational linguists refer to as entities — nouns and pronouns — with natural language processing features related to grammar errors and word usage with the aim of creating applications that can evaluate evidence of coherence in essays. View citation record >
A Developmental Writing Scale
Y. Attali & D. Powers
ETS Research Report No. RR-08-19
This report describes the development of grade norms for timed-writing performance in two modes of writing: persuasive and descriptive. View citation record >
Automated Essay Scoring With e-rater v.2.0
Y. Attali & J. Burstein
Journal of Technology, Learning, and Assessment, Vol. 4, No. 3
This article describes Version 2 of ETS's e-rater essay scoring engine. The authors present evidence on the validity and reliability of the scores that the system generates. View citation record >
Finding the WRITE Stuff: Automatic Identification of Discourse Structure in Student Essays
J. Burstein, D. Marcu, & K. Knight
IEEE Intelligent Systems: Special Issue on Advances in Natural Language Processing, Vol. 18, No. 1, pp. 32–39
In this article, the authors discuss the use of automated essay-scoring applications in the elementary through university levels for large-scale assessment and classroom instruction. View citation record >
Find More Articles
View more research publications related to automated scoring of writing quality.
Read More from Our Researchers
View a list of current ETS researchers and their work.