skip to main content skip to footer

Scoring TOEFL Essays and TOEFL 2000 Prototype Writing Tasks: An Investigation Into Raters' Decision Making and Development of a Preliminary Analytic Framework TOEFL ESL/EFL

Author(s):
Cumming, Alister H.; Kantor, Robert; Powers, Donald E.
Publication Year:
2001
Report Number:
RM-01-04, TOEFL-MS-22
Source:
ETS Research Memorandum
Document Type:
Report
Page Count:
95
Subject/Key Words:
Test of English as a Foreign Language (TOEFL), English as a Second Language (ESL), Writing (Composition), Essay Scoring

Abstract

This project established a framework to describe the decision-making processes that experienced writing assessors use to evaluate compositions written by students who speak English as a second language (ESL) or as a foreign language (EFL). The framework will assist in the development and field testing of a scoring scheme for the writing component of the new Test of English as a Foreign Language (TOEFL 2000) examination. Phase One developed empirically a) an initial framework to describe the decision-making behaviors of 10 experienced ESL/EFL instructors/assessors of differing backgrounds, each of whom produced concurrent verbal reports of their decision making while rating 60 TOEFL essays and b) a questionnaire to profile relevant variables in the raters' backgrounds. Phase Two refined the framework, gathering additional think-aloud data from a) seven highly experienced native-English-composition assessors while they rate 40 of the Phase One TOEFL essays and b) seven of the Phase One ESL/EFL instructors/assessors while they rated six TOEFL essays and 30 ESL compositions written for five TOEFL 2000 prototype tasks that involve responding to reading or listening materials. Phase Two analyses confirmed the utility of the descriptive framework and produced recommendations for the development of scoring procedures for writing tasks that integrate reading or listening materials.

Read More