skip to main content skip to footer

An Examination of Rater Orientations and Test-Taker Performance on English-for-Academic-Purposes Speaking Tasks TOEFL EAP

Author(s):
Brown, Annie; Iwashita, Noriko; McNamara, Tim
Publication Year:
2005
Report Number:
RR-05-05, TOEFL-MS-29
Source:
ETS Research Report
Document Type:
Report
Page Count:
157
Subject/Key Words:
College Board, Test of English as a Foreign Language (TOEFL), English for Academic Purposes (EAP), Oral Proficiency, Second Language Learning, Speaking Tasks, Integrated Activities

Abstract

This report documents two coordinated studies into the nature of oral English-for-academic-purposes (EAP) proficiency. Study I used verbal-report methodology to examine field experts’ rating orientations, and Study II investigated the quality of test-taker discourse on two different Test of English as a Foreign Language (TOEFL) task types (independent and integrated) at different levels of proficiency. Study I showed that, with no guidance, domain experts distinguished and described qualitatively different performances using a common set of criteria very similar to those included in draft rating scales developed for the tasks at ETS. Study II provided empirical support for the criteria applied by the judges. The findings indicate that raters take a range of performance features into account within each conceptual category and that holistic ratings are driven by all of the assessment categories rather than, as has been suggested in earlier studies, predominantly by grammar.

Read More