skip to main content skip to footer

Systems and Methods for Treatment of Aberrant Responses

Zhang, Mo; Chen, Jing; Rupp, Andre A.; Williamson, David M.
Patent Issued:
Jun 29, 2021
Patent Number:
ETS Patent
Document Type:
Family ID:
Subject/Key Words:
Patent, Active Patent, Automated Scoring, Constructed Response, Human Raters, Discrepancy Resolution, Machine Scoring


Systems and methods are provided for automatically scoring a response and statistically revaluating whether it can be considered as aberrant. In one embodiment, a constructed response is evaluated via a pre-screening stage and a post-hoc screening stage. The pre-screening stage attempts to determine whether the constructed response is aberrant based on a variety of aberration metrics and criteria. If the constructed response is deemed not to be aberrant, then the post-hoc screening stage attempts to predict a discrepancy between what score an automated scoring system would assigned and what score a human rater would assign to the response. If the discrepancy is sufficiently low, then the constructed response may be scored by an automated scoring engine. On the other hand, if the constructed response failed to pass either of the two stages, then a flag may be raised to indicate that additional human review may be needed.

Read More