Investigating the Value of TOEFL iBT Section Scores

Author(s):
Sawaki, Yasuyo; Sinharay, Sandip
Publication Year:
2013
Report Number:
RR-13-35, TOEFL-iBT-21
Source:
ETS Research Report
Document Type:
Report
Page Count:
113
Subject/Key Words:
TOEFL iBT Score Profile Reliability Test of English as a Foreign Language (TOEFL) Score Dependability Dimensionality Subscore Analysis Generalizability Theory Factor Analysis

Abstract

This study investigates the value of reporting the reading, listening, speaking, and writing section scores for the TOEFL iBT® test, focusing on 4 related aspects of the psychometric quality of the TOEFL iBT section scores: reliability of the section scores, dimensionality of the test, presence of distinct score profiles, and the section scores’ generalizability for norm-referenced decisions as well as the dependability of criterion-referenced decisions for international student admission. Four operational TOEFL iBT test forms were analyzed for all examinees as well as for 3 native language (L1) groups (Arabic, Korean, and Spanish). Haberman’s (2008) subscore analysis suggested that the speaking section score had added value due to its relative distinctness from the other modalities. Consistent with the subscore analysis results, a series of exploratory factor analyses (EFAs) indicated the possibility of the presence of 2 correlated factors—a reading/listening/writing factor and a speaking factor. In contrast, the CFAs conducted separately for the 3 L1 groups as well as a multiple-group confirmatory factor analyses (CFAs) identified a correlated 4-factor model with reading, listening, speaking, and writing factors as the best representation of the structure of the entire test for all examinees as well as for the 3 L1 groups. Reliability of the observed section scores for norm-referenced score interpretations and the dependability of classification decisions made based on different cut scores were generally satisfactory while they were also found to be relatively low in some circumstances. Based on the mixed results concerning the value-added information the TOEFL iBT section scores provide, recommendations for future research directions and some key issues of consideration for high-stakes decision making based on the section scores were summarized.

Read More