Automatic Scoring of Advanced Placement Biology Essays

Author(s):
Burstein, Jill; Kaplan, Randy; Wolff, Susanne; Lu, Chi
Publication Year:
1997
Report Number:
RR-97-22
Source:
Document Type:
Subject/Key Words:
Advanced Placement Program® automation biology computer-assisted testing constructed response tests essay tests natural language processing initiative scoring

Abstract

The natural language processing initiative (NLPI) at ETS currently does research and development for computer-assisted scoring of natural language constructed responses. The purpose of NLPI's work is to develop computer-assisted methods for scoring so that computer-administered natural language constructed-response items can be integrated into ETS' tests which can be scored efficiently with regard to time and cost. Until recently, NLPI's automated scoring efforts were primarily devoted to the development of computer programs used to score short-answer constructed-responses of up to 15 words. In previous studies, we have developed scoring systems for main idea item types (e.g., GUIDES items) and divergent thinking item types (e.g., the Formulating-Hypotheses item (FH)). In the current study, we applied similar techniques to those used to score short-answer responses in order to implement a prototype to automatically score AP® Biology essays. This paper reports on the development process involved in building a linguistically based, computer-scoring prototype for AP Biology essays and the results of the scoring system. The prototype was used to score essays that had been assigned an excellent rating (9 or 10 points) or a poor rating (0 - 3 points) by human graders. The results reported in the paper illustrate agreement between human rater scores and computer-based scores. The results of the study are promising and would encourage continued research in automated scoring of essays with highly analytical rubrics.

Read More