Automated Scoring of Math Responses

ETS's m-rater scoring engine is used for scoring open-ended mathematical responses, such as those which take the form of mathematical expressions, equations or graphs. Dating from the late 1990s, the m-rater scoring engine is one of the first ETS capabilities for automated scoring to be developed. The scores generated by the m-rater engine demonstrate very strong agreement with human ratings.

The m-rater scoring engine evaluates the correctness of a mathematical expression by determining symbolically, using a computer algebra system, if the expression is equivalent to the correct response. This enables the m-rater scoring engine to identify expressions equivalent to the key no matter what form they are found in, and to assign credit as appropriate. For instance, partial credit may be assigned if a linear equation was supposed to be provided in slope-intercept form, but was instead provided in a different, equivalent form. Scoring of mathematical responses based on string matching or text-based patterns is much more limited and error-prone than the m-rater scoring engine's capabilities for establishing equivalence symbolically.

Similarly, graph items can be scored based on a key which specifies constraints on the response entered with the graph editor. For some items, many different graphs may constitute valid answers, and the m-rater scoring engine can allow all of these variants to be scored using an elegant specification of the key.

Of course many math items are written to elicit short, text-based responses and may be more suitable for the c-rater™ engine.

Featured Publications

Below are some recent or significant publications that our researchers have authored on the subject of automated scoring of mathematical content.

2012

2011

2005

  • Online Assessment in Mathematics
    B. Sandene, R. E. Bennett, J. Braswell, & A. Oranje
    Online Assessment in Mathematics and Writing: Reports from the NAEP Technology-based Assessment Project (NCES 2005–457)
    U.S. Department of Education, National Center for Education Statistics

    The Math Online study is one of three field investigations in the National Assessment of Educational Progress Technology-Based Assessment Project, which explores the use of new technology in administering NAEP. Learn more about this publication >

Find More Articles

View more research publications related to automated scoring of math responses.