angle-up angle-right angle-down angle-left close user menu open menu closed search globe bars phone store

Automated Scoring and Natural Language Processing

Natural Language Processing (NLP)

At ETS, our researchers have extensive experience in Natural Language Processing (NLP) — a field that applies principles of linguistics and computer science to create computer applications that interact with human language.

NLP technology is the basis for the automated scoring applications that we are developing to address the increasing demand for open-ended or constructed-response test questions, which elicit responses such as extended writing responses (e.g., essays), shorter written responses to subject-matter items, and spontaneous speech. In our research, we also seek ways to build NLP technology into:

  • classroom support tools that teachers can use to help maximize the time and resources they have available for instruction
  • assessment creation tools that can aid in the test development process

Automated Scoring

ETS has been at the forefront of research in automated scoring of open-ended items for over two decades, with a long list of significant, peer-reviewed research publications as evidence of our activity in the field. ETS scientists have published on automated scoring issues in the major journals of the educational measurement, computational linguistics and language testing fields. Their work has also resulted in over 50 U.S. patents related to NLP applications for assessment, significantly more than any other organization.

The topic of automated constructed-response scoring has begun to receive substantial attention in the context of discussions related to assessment reform in the United States, as ETS measurement professionals and their colleagues in other assessment organizations noted in the recent report, Automated Scoring for the Assessment of Common Core Standards. This report highlights five key requirements to verify when considering the use of automated scoring systems:

  • Automated scores are consistent with the scores from expert human graders
  • The way automated scores are produced is understandable and substantively meaningful
  • Automated scores are fair
  • Automated scores have been validated against external measures in the same way as is done with human scoring
  • The impact of automated scoring on reported scores is understood

ETS is committed to developing automated scoring systems to meet these conditions, and evaluating them accordingly. Responsible application of automated scoring requires evaluation of all five conditions; using agreement with human raters as the sole basis for assessing the performance of a scoring system can misrepresent the effects of introducing it into large-scale operational use.

Our Research

Learn more about our research in automated scoring and natural language processing related to topics such as these:

Find More Articles

View a comprehensive list of publications related to automated scoring and natural language processing.