Automated Scoring of Speech

ETS's SpeechRater℠ engine is the world's most advanced spoken-response scoring application targeted to score spontaneous responses, in which the range of valid responses is open-ended rather than narrowly determined by the item stimulus. Test takers preparing to take the TOEFL® test have had their responses scored by the SpeechRater engine as part of the TOEFL® Practice Online test since 2006. Competing capabilities focus on assessing low-level aspects of speech production such as pronunciation by using restricted tasks in order to increase reliability. The SpeechRater engine, by contrast, is based on a broad conception of the construct of English-speaking proficiency, encompassing aspects of speech delivery (such as pronunciation and fluency), grammatical facility and higher-level abilities related to topical coherence and the progression of ideas.

The SpeechRater engine processes each response with an automated speech recognition system specially adapted for use with nonnative English. Based on the output of this system, natural language processing (NLP) and speech-processing algorithms are used to calculate a set of features that define a "profile" of the speech on a number of linguistic dimensions, including fluency, pronunciation, vocabulary usage, grammatical complexity and prosody. A model of speaking proficiency is then applied to these features in order to assign a final score to the response. While this model is trained on previously observed data scored by human raters, it is also reviewed by content experts to maximize its validity. Furthermore, if the response is found to be unscorable due to audio quality or other issues, the SpeechRater engine can set it aside for special processing.

ETS's research agenda related to automated scoring of speech includes the development of more extensive NLP features to represent pragmatic competencies and the discourse structure of spoken responses. The core capability has also been extended to apply across a range of item types used in different assessments of English proficiency, very restricted item types (such as passage read-alouds), or less restricted items (such as summarization tasks).

Featured Publications

Below are some recent or significant publications that our researchers have authored on the subject of automated scoring of speech.

2015

• Automatic Assessment of Syntactic Complexity for Spontaneous Speech Scoring
  S. Bhat & S.-Y. Yoon
  Speech Communication, Vol. 67, pp. 42–57

  The article presents a study of new measures of syntactic complexity, which automated scoring systems can use when assessing-second language spontaneous speech. Learn more about this publication >

• Feature Selection for Automated Speech Scoring
  A. Loukina, K. Zechner, L. Chen, & M. Heilman
  Paper in Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 12–19

  The authors compared different ways of selecting features in automated scoring systems that evaluate spoken or written responses in language assessments. Learn more about this publication >

• Automated Scoring of Speaking Tasks in the Test of English-for-Teaching (TEFT ™) K. Zechner, L. Chen, L. Davis, K. Evanini, C. M. Lee, C. W. Leong, X. Wang, & S.-Y. Yoon ETS Research Report No. RR-15-31

  The report summarizes research and development efforts focusing on scoring models for automatically scoring spoken-item responses, for a pilot administration of a test for nonnative English teachers or teacher candidates. Learn more about this publication >

• HALEF: An Open-Source Standard-Compliant Telephony-Based Modular Spoken Dialog System – A Review and an Outlook
  D. Suendermann-Oeft, V. Ramanarayanan, M. Teckenbrock, F. Neutatz, & D. Schmidt Paper in Proceedings of the IWSDS 2015, International Workshop on Spoken Dialog Systems, January 2015

  This paper describes completed and ongoing research on HALEF, a telephony-based open-source spoken dialog system. The system can be deployed toward a versatile range of potential applications, including intelligent tutoring, language learning and assessment. Learn more about this publication >

2014

• Performance of a Trialogue-based Prototype System for English Language Assessment for Young Learners
  K. Evanini, Y. So, J. Tao, D. Zapata-Rivera, C. Luce, L. Battistini, & X. Wang
  Paper in Proceedings of the Interspeech Workshop on Child Computer Interaction (WOCCI 2014)

  This paper describes a trialogue-based system for assessing the spoken language abilities of young learners of English. Specifically, the system employs spoken dialogue system components in interactive, conversation-based assessment tasks involving the test taker and two virtual interlocutors. Learn more about this publication >

• Automatic Detection of Plagiarized Spoken Responses
  K. Evanini & X. Wang
  Paper in Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications pp. 22–27

  This paper addresses the task of automatically detecting plagiarized responses in the context of a test of spoken English proficiency for nonnative speakers. A corpus of spoken responses containing plagiarized content was collected from a high-stakes assessment of English proficiency for nonnative speakers. Learn more about this publication >

• Similarity-Based Non-Scorable Response Detection for Automated Speech Scoring
  S. Y. Yoon & S. Xie
  Paper in Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 116–123

  This paper describes a method that filters out spoken responses from the test takers who try to game the system using diverse strategies such as speaking in their native languages or by citing memorized responses for unrelated topics. Learn more about this publication >

2013

• Automated Speech Scoring for Non-native Middle School Students with Multiple Task Types
  K. Evanini & X. Wang
  Proceedings of the 14th Annual Conference of the International Speech Communication Association (Interspeech 2013), Lyon, France, Aug. 25–29, 2013, pp. 2435–2439

  The authors present the results of applying automated speech-scoring technology to English-spoken responses provided by nonnative children in the context of an English proficiency assessment for middle school students. The challenges of using an automated spoken-language assessment for children are discussed and directions for future improvements are proposed. Learn more about this publication >

• Applying Unsupervised Learning To Support Vector Space Model Based Speaking Assessment
  L. Chen
  Conference of the North American Association for Computational Linguistics and Human Language Technologies (NAACL-HLT-2013)

  The author shows that machine-generated scores can effectively approximate the scores of human raters for use in model-building for automated speech assessment. Learn more about this publication >

• Coherence Modeling for the Automated Assessment of Spontaneous Spoken Responses
  X. Wang, K. Evanini, & K. Zechner
  Conference of the North American Association for Computational Linguistics and Human Language Technologies (NAACL-HLT)

  This paper describes a system for automatically evaluating discourse coherence in spoken responses. Learn more about this publication >

• Prompt-based content scoring for automated spoken language assessment
  K. Evanini, S. Xie, & K. Zechner
  Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, pages 157–162, Atlanta, Ga.

  This paper investigates the use of prompt-based content features for the automated assessment of spontaneous speech in a spoken language proficiency assessment. Learn more about this publication >

• Automated Content Scoring of Spoken Responses in an Assessment for Teachers of English
  K. Zechner & X. Wang
  Proceedings of the Eighth Workshop on the Innovative Use of Natural Language Processing for Building Educational Applications (BEA-8), Conference of the North American Association for Computational Linguistics and Human Language Technologies (NAACL-HLT-2013), Atlanta, Ga.

  This paper presents and evaluates approaches to automatically score the content correctness of spoken responses in a new language test for teachers of English as a foreign language who are nonnative speakers of English. Learn more about this publication >

2012

• A Comparison of Two Scoring Methods for an Automated Speech Scoring System
  X. Xi, D. Higgins, K. Zechner, & D. Williamson
  Language Testing, Vol. 29, No. 3, pp. 371–394

  In this paper, researchers compare two alternative scoring methods for an automated scoring system for speech. The authors discuss tradeoffs between multiple regression and classification tree models. Learn more about this publication >

• Exploring Content Features for Automated Speech Scoring
  S. Xie, K. Evanini, & K. Zechner
  Proceedings of the 2012 Meeting of the North American Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)
  Association for Computational Linguistics

  Researchers explore content features for automated speech scoring in this paper about automated scoring of unrestricted spontaneous speech. The paper compares content features based on three similarity measures in order to understand how well content features represent the accuracy of the content of a spoken response. Learn more about this publication >

• Assessment of ESL Learners' Syntactic Competence Based on Similarity Measures
  S. Yoon & S. Bhat
  Paper in Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)

  In this paper, researchers present a method that measures English language learners' syntactic competence for the automated speech scoring systems. The authors discuss the advantage of the current natural-language processing technique-based and corpus-based measures over the conventional ELL measures. Learn more about this publication >

2011

• Using Automatic Speech Recognition to Assess the Reading Proficiency of a Diverse Sample of Middle School Students
  K. Zechner, K. Evanini, & C. Laitusis
  Paper in Proceedings of the Interspeech Workshop on Child, Computer, and Interaction

  The authors describe a study exploring automated assessment of reading proficiency, in terms of oral reading and reading comprehension, for a middle school population including students with reading disabilities and low reading proficiency, utilizing automatic speech recognition technology. Learn more about this publication >

• A Three-Stage Approach to the Automated Scoring of Spontaneous Spoken Responses
  D. Higgins, K. Zechner, X. Xi, & D. Williamson
  Computer Speech & Language, Vol. 25, No. 2, pp. 282–306

  This paper presents a description and evaluation of SpeechRater, a system for automated scoring of nonnative speakers' spoken English proficiency. The system evaluates proficiency based on assessment tasks that elicit spontaneous monologues on particular topics. Learn more about this publication >

2009

• Automatic Scoring of Non-Native Spontaneous Speech in Tests of Spoken English
  K. Zechner, D. Higgins, X. Xi, & D. Williamson
  Speech Communication, Vol. 51, No. 10, pp. 883–895

  This paper presents the first version of the SpeechRater system, reviewing the automated scoring engine's use in the context of the TOEFL Practice Online test.
  Learn more about this publication >

Find More Articles

View more research publications related to automated scoring of speech.