Educational Applications of Natural Language Processing (NLP)

Besides scoring applications, ETS's Natural Language Processing (NLP) expertise has also resulted in other advanced capabilities to support student learning and assessment.

TextEvaluator^SM Capability

ETS's TextEvaluator^SM (formerly known as SourceRater^SM) capability represents a new approach for modeling text complexity, designed to help test developers evaluate source material for use in developing new reading comprehension passages and items. The TextEvaluator capability combines a large, cognitively based feature set with advanced psychometric techniques in order to provide text complexity classifications that are highly correlated with classifications provided by experienced educators. This feature set extends beyond the limited dimensions of text complexity assessed by other methods (such as sentence length and vocabulary) to encompass text-level cohesion and account for differences between different text genres.

Language Muse^SM Application

Language Muse^SM is a web-based, instructional authoring application intended to support K–12 teachers in the development of curriculum for English language learners (ELLs). The application offers linguistic feedback that highlights vocabulary, sentence structures and discourse relations found in classroom texts that may be unfamiliar to ELLs. The linguistic feedback supports teachers in creating linguistically-informed lesson plans, texts, activities and assessments with appropriate scaffolding. The Language Muse application has been used in formal teacher professional development settings to help teachers cultivate linguistic awareness so that they are better able to create a curriculum that addresses students' English language learning needs. The application contains self-guided professional development, so teachers can complete that portion on their own and continue to use the application in the classrooms to more easily design scaffolded materials appropriate to every K–12 grade level.

Automated Test Item Generation

Another area in which ETS has applied its natural language processing technology is in the automated generation of test items. This includes research both on completely automated generation of items from item models (in order to reduce the cost of item development and control item difficulty) and semi-automated item creation tools to help assessment developers identify appropriate source material for items or create draft items that can be augmented and edited by experienced item writers.

Featured Publications

Below are some recent or significant publications that our researchers have authored on the subject of educational applications of natural language processing technology.

2012

• Difficulty Modeling and Automatic Generation of Quantitative Items: Recent Advances and Possible Next Steps
  E. A. Graf & J. H. Fife
  Chapter in Automatic Item Generation: Theory and Practice, pp. 157–180
  Editors: M. Gierl & T. Haladyna
  Publisher: Routledge

  This ETS-authored chapter is part of a book volume that aims to summarize current knowledge about the field of automatic item generation. The chapter appears in Part III of the volume, which covers psychological and substantive characteristics of generated items. Read more on the publisher's website.

• Item Generation: Implications for a Validity Argument
  I. Bejar
  Chapter in Automatic Item Generation: Theory and Practice, pp. 40–56
  Editors: M. Gierl & T. Haladyna
  Publisher: Routledge

  This ETS-authored chapter is part of a book volume that aims to summarize current knowledge about the field of automatic item generation. The chapter appears in Part I of the volume, which covers initial considerations for automatic item generation. Read more on the publisher's website.

2010

• The Utility of Article and Preposition Error Correction Systems for English Language Learners: Feedback and Assessment
  M. Chodorow, M. Gamon, & J. Tetreault
  Language Testing, Vol. 27, No. 3, pp. 419–436

  In this article, the authors describe and evaluate two systems for identifying and correcting writing errors involving English articles and prepositions. View the full abstract or order this article.

• Automated Grammatical Error Detection for Language Learners
  C. Leacock, M. Chodorow, M. Gamon, & J. Tetreault
  Monograph in Synthesis Lectures on Human Language Technologies
  Morgan & Claypool

  This volume describes the types of constructions English language learners find most difficult — constructions containing prepositions, articles and collocations — and it provides an overview of the automated approaches to identifying and correcting such learner errors. View the full abstract or order this report.

• Using an Error-Annotated Learned Corpus to Develop an ESL/EFL Error Correction System
  N.-R. Han, J. Tetreault, S.-H. Lee, & J.-Y. Ha
  Proceedings of the International Conference on Language Resources and Evaluation (LREC 2010)
  European Language Resources Association

  This paper presents research on building a model of grammatical error correction, for preposition errors in particular, in English text produced by English language learners. Download the full report.

2009

• Opportunities for Natural Language Processing in Education
  J. Burstein
  Computational Linguistics and Intelligent Text Processing 10th International Conference, CICLing 2009, Mexico City, Mexico, March 1–7, 2009. Proceedings
  Springer

  This paper discusses emerging opportunities for natural language processing researchers in the development of educational applications for writing, reading and content knowledge acquisition. View the full abstract or order this report.

2008

• The Ups and Downs of Prepositional Error Detection in ESL Writing
  J. Tetreault & M. Chodorow
  Proceedings of the 22nd International Conference on Computational Linguistics, pp. 865–872
  Association for Computational Linguistics

  This paper describes a method of detecting preposition errors in the writing of nonnative English speakers. It also discusses current approaches to annotation and evaluation. Download the full report.

• A Computational Approach to Detecting Collocation Errors in the Writing of Non-native Speakers of English
  Y. Futagi, P. Deane, M. Chodorow, & J. Tetreault
  Computer Assisted Language Learning, Vol. 21, pp. 353–367

  This paper describes the first prototype of an automated tool for detecting collocation errors in texts written by nonnative speakers of English. View the full report.

• When Do Standard Approaches for Measuring Vocabulary Difficulty, Syntactic Complexity and Referential Cohesion Yield Biased Estimates of Text Difficulty?
  K. M. Sheehan, I. Kostin, & Y. Futagi
  Paper in Proceedings of the 30th Annual Meeting of the Cognitive Science Society

  This paper demonstrates that many widely-used approaches for assessing text difficulty tend to both overpredict the difficulty of informational texts and underpredict the difficulty of literary texts. Download the full report.

2007

• The Automated Text Adaptation Tool
  J. Burstein, J. Shore, J. Sabatini, Y. Lee, & M. Ventura
  Proceedings of Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pp. 3–4
  Association for Computational Linguistics

  This paper introduces the Automated Text Adaptation Tool v.1.0 (ATA v.1.0), an innovative, educational tool that automatically generates text adaptations similar to those teachers might create. Download the full report.

• Item Distiller: Text Retrieval for Computer-Assisted Test Item Creation
  D. Higgins
  ETS Research Memorandum No. RM-07-05

  This paper describes Item Distiller, a tool developed at ETS to aid in the creation of sentence-based test items. View the full abstract.

• SourceFinder: A Construct-Driven Approach for Locating Appropriately Targeted Reading Comprehension Source Texts
  K. M. Sheehan, I. Kostin, & Y. Futagi
  Proceedings of the 2007 Workshop of the International Speech Communication Association, Special Interest Group on Speech and Language Technology in Education

  This paper describes an automated approach for locating source material for use in developing reading passages. Download the full paper.

2006

• Model Analysis and Model Creation: Capturing the Task-Model Structure of Quantitative Item Domains
  P. Deane, E. A. Graf, D. Higgins, Y. Futagi, & R. Lawless
  ETS Research Report No. RR-06-11

  This study focuses on the relationship between item modeling and evidence-centered design (ECD); it considers how an appropriately generalized item modeling software tool can support systematic identification and exploitation of task-model variables, and then examines the feasibility of this goal, using linear-equation items as a test case. View the full abstract.