Educator Series Connections

ETS Educator Series Connections

Praxis® Program to Use Automated Scoring for Additional Constructed-response Tests

Since 2014, the Praxis® program has used the e-rater® scoring engine for one of the two essays included in the Praxis® Core Academic Skills for Educators (Core) Writing test. Beginning in Fall of 2017, the e-rater engine will be used for the scoring of the second essay as well, and additional Praxis constructed-response tests will be scored using another automated scoring technology developed at ETS.

ETS began conducting research on automated scoring of constructed-response tasks in the 1980s and expanded this research to incorporate Natural Language Processing (NLP) technologies in the mid- 1990s. This line of research ultimately resulted in multiple types of automated scoring technologies for such fields and areas as essays, mathematics and spoken responses.

The e-rater engine has been in operational use since 1999, and is used to assist human raters in scoring essays on the GRE® General Test, the TOEFL® test, and the Praxis Core Writing test. The e-rater engine computes a score based on data from hundreds of previously scored essays. It estimates the score for an essay based on features related to writing quality, including grammar, usage, mechanics, style, organization and development.

Over the years, ETS researchers have extended their work in NLP research and developed other automated scoring technologies: the c-rater™ engine, the m-rater engine, and the SpeechRaterSM engine. ETS has incorporated these technologies into many of its testing programs, products, and services, including the Keeping Learning on Track® program, the Criterion® Online Writing Evaluation Service, and TOEFL® Practice Online.

For the Praxis Subject Assessments, the automated scoring engine that will be employed is c-rater, which is ETS's system for the automated scoring of content. The technology is designed to score items that elicit specific information regardless of the grammatical or stylistic form chosen by the author. This makes the c-rater system ideal for evaluating content for subject-matter constructed-response questions in many areas, including science, reading comprehension and mathematics. The c-rater engine uses machine-learning techniques to create scoring models for automatically assigning scores to candidate responses. This approach does not require all possible correct responses to be provided to the system. Instead, an appropriate set of responses to the item that have already been holistically scored by trained raters is required.

The Praxis tests that may be scored by c-rater include Elementary Education: Instructional Practice and Applications (5019), Education of Young Children (5024), English Language Arts: Content and Analysis (5039), Middle School English Language Arts (5047), Social Studies: Content and Interpretation (5086), Middle School Social Studies (5089), Physical Education: Content and Design (5095), Teaching Reading: Elementary Education (5203), Teaching Reading (5204), Reading Specialist (5301), Reading for Virginia Educators: Reading Specialist (5304), Reading for Virginia Educators: Elementary and Special Education (5306), Special Education: Core Knowledge and Mild to Moderate Applications (5543), Special Education: Core Knowledge and Severe to Profound Applications (5545), World Languages Pedagogy (5841), the School Superintendent Assessment (6021), and the four Principles of Learning and Teaching tests (5621, 5622, 5623, and 5624).

For the tests that use automated scoring, each response will receive a score from at least one trained reader, using a holistic scoring scale, and will then receive a score from the automated scoring engine. In holistic scoring, readers are trained to assign scores on the basis of the overall quality of a response to the assigned task. If the score generated by the automated scoring engine and the human score agree, the two scores are added to become the final score for the response. If they disagree by more than a certain amount, a second human rater scores the response, and the scores that are in agreement are added to become the final score for the response.

For more information about the e-rater and c-rater engines, visit About the e-rater Scoring Engine and Automated Scoring of Written Content, and for further information on NLP, visit Automated Scoring and Natural Language Processing.

Back to Top

ETS Welcomes Monica A. Beane to its Client Relations Team

Monica A. Beane has joined ETS as a Client Relations Director. In her new role, she will serve as the first line of contact for our clients in Alaska, California, Hawaii, Idaho, Montana, Nevada, Utah and Washington.

Monica brings over 20 years of education experience, including posts as an executive director of the Office of Educator Effectiveness and Licensure at the West Virginia Department of Education, an elementary school principal, a middle school English and reading teacher, an elementary school teacher, and a preschool special needs teacher in North Carolina and West Virginia public schools. In her most recent position as Executive Director of the Oregon Teacher Standards and Practices Commission, Monica was charged with continuing the Commission's commitment to equity, implementing the legislatively approved licensure redesign, updating professional practice standards and overseeing the movement to national accreditation for all Oregon educator preparation programs.

A passionate supporter of education, Monica sees enhancing the continuum of educator development from pre-service to professional growth as an essential piece in addressing teacher shortages. Strong preparation and support increases teachers' efficacy and makes it more likely they will remain in the profession. Building value and elevating the education profession also serves to attract new candidates. She sees effective educator development as one with "minimum number of barriers at the highest level of validity."

Monica holds a doctorate in educational leadership, is a National Board Certified Teacher (NBCT), and is a three-time recipient of Arch Coal's Golden Apple Achievement Award for excellence in teaching.

Outside of work, Monica takes pleasure in spending time with her family and enjoys being outdoors.

Please join us in welcoming Monica to ETS.

Back to Top