Educational Applications of Natural Language Processing (NLP)

Besides scoring applications, ETS's Natural Language Processing (NLP) expertise has also resulted in other advanced capabilities to support student learning and assessment.

Writing Mentor^™

The Writing Mentor^™ application is a Google Docs writing support add-on. The app targets a wide range of postsecondary users, including struggling writers and English learner (EL) populations enrolled in 2- and 4-year colleges. The app is intended to provide one-stop-shopping for writers who are looking for some writing help. Students who are using Google Docs can install the app and use it to get feedback for text — specifically, actionable feedback about their writing related to claims and sources, topic development, coherence, and English conventions and word choice. Feedback leverages ETS's natural language processing (NLP) capabilities and lexical resources, and synonyms for unfamiliar words they may encounter while reading external sources. In addition to feedback, the app provides a report illustrating the different feedback types that the user viewed. The report can be saved as a PDF file to show to their instructor. It can give the instructor a sense of how their students may be engaging with the tool, and what aspects of writing they are working on.

The Language Muse^® Activity Palette

The Language Muse^® Activity Palette is a web-based application designed to support English Learners (ELs). Aligned with reading standards, the tool automatically generates customizable activities aimed to help ELs build the academic language skills needed for deeper reading comprehension in content areas. The language-based activities are intended to support content comprehension and language skills development through activities that afford practice with vocabulary, sentence structures, discourse and summary writing. Teachers can use the tool to create and administer a "palette" of online activities for classroom texts that students can complete, and are scored online. Paper-and-pencil assignments are also available. Activities can be used for classroom discussion, independent or group work. While the tool targets ELs, activities may be useful for all students. Teachers can use their own texts, or the library of texts provided with the tool. This work has been funded by the Institute of Education Sciences (IES), United States Department of Education (R305A140472).

TextEvaluator^® Capability

TextEvaluator^® is a fully-automated, web-based technology tool designed to help teachers, textbook publishers and test developers select texts for use in instruction and assessment. TextEvaluator incorporates a patented measurement approach that goes beyond traditional readability dimensions of syntactic complexity and vocabulary difficulty to address complexity variation due to cohesion, concreteness, academic orientation, level of argumentation, degree of narrativity and degree of adherence to an interactive/conversational style. Resulting feedback about the complexity characteristics of texts can be expressed at either of two levels of granularity:

• as a single, overall text complexity score aligned with the grade-by-grade text complexity guidelines specified in the Common Core State Standards
• as a profile of eight component scores constructed to highlight the aspects of text variation expected to be most challenging within any specified text

Featured Publications

Below are some recent or significant publications that our researchers have authored on the subject of educational applications of natural language processing technology.

2017

• Generating Language Activities in Real-Time for English Learners Using Language Muse^®
  J. Burstein, N. Madnani, J. Sabatini, D. McCaffrey, K. Biggers, & K. Dreier. Proceedings of the Fourth Annual ACM Conference on Learning at Scale, Cambridge, MA.

  The paper describes the Language Muse^® Activity Palette, a web-based language-instruction application that uses NLP algorithms and lexical resources to automatically generate language activities and support English language learners' content comprehension and language skills development. Pilot studies conducted using the application are discussed as well.

2016

• A Review of Evidence Presented in Support of Three Key Claims in the Validity Argument for the TextEvaluator^® Text Analysis Tool
  K. M. Sheehan
  ETS Research Report No. RR-16-12

  This paper provides an overview of the TextEvaluator^® measurement approach and summarizes evidence related to three key claims in the TextEvaluator validity argument. View the citation.

• Language Muse^® Automated Linguistic Activity Generation for English Language Learners
  N. Madnani, J. Burstein, J. Sabatini, K. Biggers & S. Andreyev
  Proceedings of the Annual Meeting of the Association for Computational Linguistics, Berlin, Germany. pp. 79–84

  This paper describes the features and functionality of Language Muse^®, a web-based tool that uses NLP algorithms to automatically generate customizable linguistic activities. View the article >

• The Language Muse^® Activity Palette: Technology for Promoting Improved Content Comprehension for English Language Learners
  J. Burstein, J. Sabatini, S.A. Crossley & D.S. McNamara
  Adaptive Educational Technologies for Literacy Instruction. Taylor & Francis, Routledge: NY.

  This book chapter highlights the usefulness of the Language Muse^® Activity Palette as a tool for aiding English Language Learners (ELLs). The palette can be used to generate language activities for many purposes, but many of its features were designed with the ELL population in mind. Access the complete book >

2015

• Aligning TextEvaluator^® Scores With the Accelerated Text Complexity Guidelines Specified in the Common Core State Standards
  K. M. Sheehan
  ETS Research Report No. RR-15-21

  This paper describes the innovative approach used to align the TextEvaluator^® scale with the text complexity scale specified in the Common Core State Standards. View citation record >

• Using TextEvaluator^® to Quantify Sources of Linguistic Complexity in Textbooks Targeted at First Grade Readers Over the Past Half Century
  K. M. Sheehan, M. Flor, D. Napolitano, & C. Ramineni
  ETS Research Report No. RR-15-38

  This paper demonstrates that, in contrast to the oft-repeated claim of a general steady decline in textbook complexity, text-based sources of comprehension difficulty within Grade 1 textbooks have either risen or held steady throughout the past half century. View the full report >

• Analyzing and Comparing Reading Stimulus Materials Across the TOEFL^® Family of Assessments
  J. Chen & K. M. Sheehan
  TOEFL iBT Research Report No. 26 and ETS Research Report No. RR-15-08

  This paper compares the linguistic complexity of passages selected from the reading sections of three different assessments: the TOEFL^® Primary™ assessment, the TOEFL Junior^® assessment and TOEFL iBT^® assessment. Implications with respect to the goal of selecting new passages that are optimally structured for use on one or another of these three assessments are discussed. View the full report >

• Online Readability and Text Complexity Analysis with TextEvaluator^®
  D. Napolitano, K. M. Sheehan & R. Mundkowsky
  Proceedings of Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pp. 96–100
  Association for Computational Linguistics

  In this paper, the authors provide an overview of how a person can obtain and interpret TextEvaluator^® analysis received via the Web. View the full article >

2014

• From Teacher Professional Development to the Classroom: How NLP Technology Can Enhance Teachers' Linguistic Awareness to Support Curriculum Development for English Language Learners
  J. Burstein, J. Shore, J. Sabatini, B. Moulder, J. Lentini, K. Biggers, & S. Holtzman
  Journal of Educational Computing Research, 51(1): 119–144

  This article discusses early work and pilot studies about the Language Muse Activity Palette. View the article >

• The TextEvaluator^® Tool: Helping Teachers and Test Developers Select Texts for Use in Instruction and Assessment
  K. M. Sheehan, I. Kostin, D. Napolitano, & M. Flor
  The Elementary School Journal, Vol. 115, No. 2, pp. 184–209

  This article describes TextEvaluator^®, a comprehensive text-analysis system designed to help teachers, textbook publishers, test developers and literacy researchers select reading materials that are consistent with the text complexity goals outlined in the Common Core State Standards. View citation record >

2013

• Measuring Cohesion: An Approach That Accounts for Differences in the Degree of Integration Challenge Presented by Different Types of Sentences
  K. M. Sheehan
  Educational Measurement: Issues and Practice, v32 n2 pp. 28–37, Win 2013

  This article will first review previous cohesion research, distinguishing between studies focused on the validity of proposed metrics, and studies that merely summarize the cohesion scores obtained for different types of texts and then will describe two general approaches for measuring cohesion. View the full article >

• A Two-Stage Approach for Generating Unbiased Estimates of Text Complexity
  K. M. Sheehan, M. Flor, & D. Napolitano
  Proceedings of the Second Workshop on Natural Language Processing for Improving Textual Accessibility (NLP4ITA), pp. 49–58, Atlanta, Ga. Association for Computational Linguistics.

  This paper presents a two-stage estimation technique that successfully addresses the tendency of automated text complexity tools to overestimate the complexity levels of informational texts, while simultaneously underestimating the complexity levels of literary texts. View citation record >

• A User Study: Technology to Increase Teachers' Linguistic Awareness to Improve Instructional Language Support for English Language Learners
  J. Burstein, J. Sabatini, J. Shore, B. Moulder, & J. Lentini
  In Proceedings of the Workshop for Improving Textual Accessibility in conjunction with the Annual Meeting of the North American Association for Computational Linguistics, Atlanta, Ga., June 14, 2013

  This paper discusses user study outcomes with teachers who used LanguageMuse, a web-based teacher professional development (TPD) application designed to enhance teachers' linguistic awareness, and support teachers in the development of language-based instructional scaffolding (support) for their English-language learners (ELL). View citation record >

• Lexical Tightness and Text Complexity
  M. Flor, B. Beigman Klebanov & K. M. Sheehan
  In Proceedings of the 2nd Workshop of Natural Language Processing for Improving Textual Accessibility (NLP4ITA), 2013, pp. 29–38

  This paper presents our methodology for building word association profiles for texts, it defines the measure of lexical tightness (LT) and describes the datasets used in the study, it presents our study of the relationship between LT and text complexity, describes application to poetry, evaluates an improved measure (LTR) and reviews related work. View the full paper >

2012

• A Cognitively-Based Text Analysis System Designed to Help Test Developers Ensure that Admissions Assessments Incorporate Suitably Complex Texts
  K. M. Sheehan
  Proceedings of the First International Conference on Assessment & Evaluation, Admission Criteria in Higher Education, Volume 1, English Proceedings, Riyadh, Saudi Arabia, Dec. 2–4, 2012, pp. 43–58

  This paper provides an overview of TextEvaluator's innovative estimation approach, and then highlight one particularly useful aspect of this new system: an approach for measuring the ease or difficulty of required referential and connective inferences. View the article >

2010

• Generating Automated Text Complexity Classifications that are Aligned with Targeted Text Complexity Standards
  K. M. Sheehan, I. Kostin, Y. Futagi & M. Flor
  ETS Research Report No. RR-10-28

  Three approaches for generating improved measures of text complexity are discussed: expanding construct coverage, selecting more appropriate criterion scores, and accounting for genre effects. A text complexity measure that incorporates all three improvements is introduced. Validity analyses suggest that text complexity classifications generated via the proposed tool are closely aligned with complexity classifications provided by human experts. View full report >

• Automated Grammatical Error Detection for Language Learners
  C. Leacock, M. Chodorow, M. Gamon, & J. Tetreault
  Monograph in Synthesis Lectures on Human Language Technologies
  Morgan & Claypool

  This volume describes the types of constructions English-language learners find most difficult — constructions containing prepositions, articles and collocations — and it provides an overview of the automated approaches to identifying and correcting such learner errors. View citation record >

2009

• Opportunities for Natural Language Processing in Research Education
  J. Burstein
  Computational Linguistics and Intelligent Text Processing 10th International Conference, CICLing 2009, Mexico City, Mexico, March 1–7, 2009. Proceedings
  Springer

  This paper discusses emerging opportunities for natural language-processing researchers in the development of educational applications for writing, reading and content knowledge acquisition. View citation record >

2008

• When Do Standard Approaches for Measuring Vocabulary Difficulty, Syntactic Complexity and Referential Cohesion Yield Biased Estimates of Text Difficulty?
  K. M. Sheehan, I. Kostin, & Y. Futagi
  Paper in Proceedings of the 30th Annual Meeting of the Cognitive Science Society

  This paper demonstrates that many widely used approaches for assessing text difficulty tend to both overpredict the difficulty of informational texts and underpredict the difficulty of literary texts. View citation record >

2007

• The Automated Text Adaptation Tool
  J. Burstein, J. Shore, J. Sabatini, Y. Lee, & M. Ventura
  Proceedings of Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pp. 3–4
  Association for Computational Linguistics

  This paper introduces the Automated Text Adaptation Tool v.1.0 (ATA v.1.0), an innovative, educational tool that automatically generates text adaptations similar to those teachers might create. View citation record >

• SourceFinder: A Construct-Driven Approach for Locating Appropriately Targeted Reading Comprehension Source Texts
  K. M. Sheehan, I. W. Kostin, & Y. Futagi
  Proceedings of the 2007 Workshop of the International Speech Communication Association, Special Interest Group on Speech and Language Technology in Education, pp. 80–83

  This paper describes a fully-automated approach for locating source material for use in developing reading comprehension/verbal reasoning passages. View citation record >

Find More Articles

View more research publications related to educational applications of natural language processing.