English Language Learning and Assessment Worldwide

ETS has a long history of supporting research on global assessments for adult and older adolescent English learners who need to demonstrate English proficiency for academic or job-related purposes. Our research on assessments for younger learners in recent years has provided a strong research foundation for newly introduced assessments that target middle school students in global contexts — the TOEFL® Junior™ Standard and Comprehensive tests.  
Our research in this area has aims such as these:

• Validity and Fairness Frameworks — We conduct foundational research with the aim of providing conceptual frameworks to guide the practice of developing fair and valid assessments.
• Test Design Frameworks and Principles — We develop new test design frameworks and principles that are theoretically-grounded and practically applicable.
• Quality — We maintain an ongoing program of research to support and continuously improve existing testing programs. As part of this focus on quality, we also conduct foundational research with the aim of understanding language development, identifying factors that may affect test-taker performances, understanding rater behaviors and improving practices in test scoring. 
• Innovation — We seek to design new measures, in particular those that utilize new methods and language technologies to support English learning and instruction.

Research Related to ETS Testing Programs

• The TOEFL iBT® test — Learn more about the research we conduct to support this assessment of students' ability to use English for communication at the university level across the globe.
• The TOEIC® test — Learn more about the research we conduct to support this assessment of test-takers' ability to use English for communication in the global workplace.

Foundational Research

In addition to the publications that our research professionals have authored in direct support of the TOEFL® and TOEIC tests, our English Language Learning and Assessment research initiative publishes or funds works related to foundational research in an international context. Topics of such research include validity and fairness frameworks; test design frameworks and principles; quality; and innovation. Below are some recent publications:

Validity and Fairness Frameworks

• Does an Argument-Based Approach to Validity Make a Difference?
  C. A. Chapelle, M. K. Enright, & J. Jamieson (2010)
  Educational Measurement: Issues and Practice, Vol. 29, No. 1, pp. 3–13
  
  This paper evaluates the differences between two different approaches to validity including Kane (2006) and the 1999 AERA/APA/NCME Standards for Educational and Psychological Testing. Read more or order the article from the publisher.
• Articulating a Validity Argument
  M. Kane (2012)
  Chapter in The Routledge Handbook of Language Testing, pp. 34–47
  Editors: G. Fulcher & F. Davidson
  Publisher: Routledge
  
  In this book chapter, the author discusses an argument-based approach to validation and provides examples in the context of language testing. Learn more about the book in which this publication appears.
• Validating Score Interpretations and Uses
  M. Kane (2012)
  Language Testing, Vol. 29, No. 1, pp. 3–17
  
  In this article, the author discusses a two-step, argument-based approach to validation involving (1) specifying the proposed uses and interpretations of the test’s scores, and (2) evaluating the plausibility of the proposed interpretive argument. Read more or order the article from the publisher.
  
• A Framework for Evaluation and Use of Automated Scoring
  D. Williamson, X. Xi & J. Breyer (2012)
  Educational Measurement: Issues and Practice, Vol. 31, No. 1, pp. 2–13
  
  This article provides a framework for evaluating and using automated scoring for constructed-response tasks. The framework entails both evaluation of automated scoring as well as guidelines for implementing and maintaining it in the context of constantly evolving technologies. Read more or order the article from the publisher.
• Validity and the Automated Scoring of Performance Tests
  X. Xi (2012)
  Chapter in The Routledge Handbook of Language Testing, pp. 438–451
  Editors: G. Fulcher & F. Davidson
  Publisher: Routledge
  
  In this book chapter, the author discusses validity considerations in using automated scoring for performance-based language tests in the context of evolving theories and practice in test validity. Learn more about the book in which this publication appears.
• How Do We Go About Investigating Test Fairness?
  X. Xi (2010)
  Language Testing, Vol. 27, No. 2, pp. 147–170
  
  This article proposes an approach that treats fairness as an aspect of validity and provides an illustration of how a fairness argument may be established and supported in a validity argument. Read more or order the article from the publisher.
  
• Methods of Test Validation
  X. Xi (2008)
  Chapter in Encyclopedia of Language and Education, Volume 7: Language Testing and Assessment, 2nd fully revised edition (pp. 177–196).
  Editors: E. Shohamy & N. H. Hornberger
  Publisher:  Springer
  
  This chapter provides a comprehensive examination of the evolution of the concept of validity and presents current validation methods for language assessments. The author also discusses how advances in validity research in language assessment benefit from progress in other fields. Read more or order this volume from the publisher. 
• What and How Much Evidence Do We Need? Critical Considerations in Validating an Automated Scoring System
  X. Xi (2008)
  Chapter in Towards Adaptive CALL: Natural Language Processing for Diagnostic Language Assessment (pp. 102–114)
  Editors: C. A. Chapelle, Y.-R. Chung, & J. Xu
  Publisher: Iowa State University
  
  This paper illustrates how an argument-based approach can be applied to the validation of the use of an automated scoring system called SpeechRater^SM for the TOEFL® Practice Online Speaking test.  Download the full paper.
  

Test Design Frameworks and Principles

• Evidence-Centered Design in Language Testing
  R. J. Mislevy & C. Yin (2012)
  Chapter in The Routledge Handbook of Language Testing, pp. 208–222
  Editors: G. Fulcher & F. Davidson
  Publisher: Routledge
  
  Evidence-centered design (ECD) is a conceptual framework for the design and delivery of assessments. This book chapter discusses the ways ECD can be used effectively in language testing. Learn more about the book in which this publication appears.
• Prototyping New Item Types
  S. Nissan & M. Schedl (2012)
  Chapter in The Routledge Handbook of Language Testing, pp. 281–294
  Editors: G. Fulcher & F. Davidson
  Publisher: Routledge
  
  This book chapter offers a definition of prototyping, considers the ideal characteristics of a prototyping population, and discusses the kinds of information that prototyping can provide for the design and development of new assessments. Learn more about the book in which this publication appears.
• The Case for a Comprehensive, Four-Skills Assessment of English-Language Proficiency
  D. E. Powers (2010)
  ETS R&D Connections No. 14
  
  This article makes a case for measuring a test-taker's overall proficiency in all modes of communication in English, including listening, reading, writing and speaking. Read more or download the full article.

Quality

• The Effectiveness of Feedback for L1-English and L2-Writing Development: A Meta-Analysis
  D. Biber, T. Nekrasova, & B. Horn (2011)
  ETS Research Report No. RR-11-05
  
  This report reviews and synthesizes previous research on the effectiveness of feedback for individual writing development. The meta-analysis of research in this area shows that feedback is beneficial for writing development. Read more or download the full report.
• Reverse Linguistic Stereotyping: Measuring the Effect of Listener Expectations on Speech Evaluation
  O. Kang & D. Rubin (2009)
  Journal of Language and Social Psychology, Vol. 28, No. 4, pp. 441–456
  
  This article investigates a phenomenon known as reverse linguistic stereotyping, in which attributions of a speaker’s group membership trigger distorted evaluations of that person’s speech. Read more or order this article from the publisher.
• Suprasegmental Measures of Accentedness and Judgments of Language Learner Proficiency in Oral English
  O. Kang, D. Rubin, & L. Pickering (2010)
  Modern Language Journal, Vol. 94, No. 4, pp. 554–566
  
  This study examines the relationships among several acoustic measures of accented speech and native listeners’ judgments of oral proficiency. The suprasegmental features of speech were found to be strong predictors of oral proficiency and comprehensibility. Read more or order the article from the publisher.
  
• Aspects of Performance on Line Graph Description Tasks: Influenced by Graph Familiarity and Different Task Features
  X. Xi (2010)
  Language Testing, Vol. 27, No. 1, pp. 73–100
  
  This article describes a study that systematically manipulated characteristics of a line graph description task in a speaking test with the aim of mitigating the influence of graph familiarity, a potential source of construct-irrelevant variance in testing. Read more or order the article from the publisher.

Innovation

• The Utility of Article and Preposition Error Correction Systems for English Language Learners: Feedback and Assessment
  M. C. Chodorow, M. Gamon, & J. R. Tetreault (2010)
  Language Testing, Vol. 27, No. 3, pp. 419–436
  
  This paper describes and evaluates two systems for identifying and correcting writing errors involving English articles and prepositions. Results show that both systems were helpful for error corrections. Read more or order the article from the publisher.
  
• A Computational Approach to Detecting Collocation Errors in the Writing of Non-Native Speakers of English 
  Y. Futagi, P. Deane, M. Chodorow, & J. Tetreault (2008)
  Computer Assisted Language Learning, Vol. 21, No. 4, pp. 353–367
  
  This article describes a prototype of an automated tool for detecting collocation errors in the writing of English learners. Detailed error analyses and possible improvements of the system are discussed. Read more or order the article from the publisher.
• Automated Grammatical Error Detection for Language Learners
  C. Leacock, M. Chodorow, M. Gamon, & J. Tetreault (2010)
  Synthesis Lectures on Human Language Technologies, Vol. 3, No. 1, pp. 1–134
  
  This volume provides an overview of automated approaches that have been developed to identify and correct different types of grammatical errors in a number of languages. New directions for research in automated grammatical error detection are proposed. Read more or order this publication from the publisher.
  
• Automated Scoring and Feedback Systems: Where Are We and Where Are We Heading?
  X. Xi (2010)
  Language Testing, Vol. 27, No. 3, pp. 291–300
  
  This editorial prefaces a special edition of Language Testing that provides a collection of new methodologies and approaches in automated scoring and feedback systems. Background information and issues related to automated scoring and automated feedback research are discussed. Read more or order this article from the publisher.
• Adapting the Acoustic Model of a Speech Recognizer for Varied Proficiency Non-Native Spontaneous Speech Using Read Speech with Language-Specific Pronunciation Difficulty
  K. Zechner, D. Higgins, R. Lawless, Y. Futagi, S. Ohls, & G. Ivanov (2009)
  Paper in Proceedings of Interspeech 2009: 10th Annual Conference of the International Speech Communication Association, Vol. 1–5, pp. 612–615
  
  This paper presents an approach to acoustic model adaptation of a recognizer for nonnative spontaneous speech in the context of recognizing candidates’ responses in a test of spoken English. Read more or download the full paper.