Scoring TOEFL® Essays and TOEFL® 2000 Prototype Writing Tasks: An Investigation Into Raters' Decision Making and Development of a Preliminary Analytic Framework

Author(s):
Cumming, Alister; Kantor, Robert; Powers, Donald E.
Publication Year:
2001
Report Number:
RM-01-04, TOEFL-MS-22
Source:
Document Type:
Subject/Key Words:
Assessment of ESL writing rating processes think-aloud verbal reports TOEFL® background influences on assessment evaluating compositions written from sources

Abstract

This project established a framework to describe the decision-making processes that experienced writing assessors use to evaluate compositions written by students who speak English as a second language (ESL) or as a foreign language (EFL). The framework will assist in the development and field-testing of a scoring scheme for the writing component of the new TOEFL® 2000 test (Test of English as a Foreign Language™) examination. Phase One developed empirically (a) an initial framework to describe the decision-making behaviors of 10 experienced ESL/EFL instructors/assessors of differing backgrounds, each of whom produced concurrent verbal reports of their decision making while rating 60 TOEFL essays and (b) a questionnaire to profile relevant variables in the raters' backgrounds. Phase Two refined the framework and questionnaire, gathering additional think-aloud data from seven highly experienced native-English-composition assessors while they rated 40 of the Phase One TOEFL essays and 30 ESL compositions written for five TOEFL 2000 prototype tasks that involve responding to reading or listening materials. Phase Two analyses confirmed the utility of the descriptive framework and produced recommendations for the development of scoring procedures for writing tasks that integrate reading or listening materials.

Read More