Automated Scoring and Natural Language Processing
Natural Language Processing (NLP)
At ETS, our researchers have extensive experience in natural language processing (NLP) — a field that applies principles of linguistics and computer science to create computer applications that interact with human language.
NLP technology is the basis for the automated scoring applications that we are developing to address the increasing demand for open-ended or constructed-response test questions, which elicit responses such as short written answers, essays, recorded speech, mathematical equations or graphs. In our research, we also seek ways to build NLP technology into:
- classroom support tools that teachers can use to help maximize the time and resources they have available for instruction
- assessment creation tools that can aid in the test development process
ETS has been at the forefront of research in automated scoring of open-ended items for over two decades, with a long list of significant, peer-reviewed research publications as evidence of our activity in the field. ETS scientists have published on automated scoring issues in the major journals of the educational measurement, computational linguistics and language testing fields. Their work has also resulted in 19 U.S. patents related to applying NLP in assessment, significantly more than any other organization.
The topic of automated constructed-response scoring has begun to receive substantial attention in the context of discussions related to assessment reform in the United States, as ETS measurement professionals and their colleagues in other assessment organizations noted in the recent report, Automated Scoring for the Assessment of Common Core Standards. This report highlights five key requirements to verify when considering the use of automated scoring systems:
- Automated scores are consistent with the scores from expert human graders
- The way automated scores are produced is understandable and substantively meaningful
- Automated scores are fair
- Automated scores have been validated against external measures in the same way as is done with human scoring
- The impact of automated scoring on reported scores is understood
ETS is committed to developing automated scoring systems to meet these conditions, and evaluating them accordingly. Responsible application of automated scoring requires evaluation of all five conditions; using agreement with human raters as the sole basis for assessing the performance of a scoring system can misrepresent the effects of introducing it into large-scale operational use.
Learn more about our research in automated scoring and natural language processing related to topics such as these:
A comprehensive, reverse-chronological bibliography of our NLP and automated-scoring related work is also available.