A Study of Characteristics of the SPEAK Test

Author(s):
Sarwark, Susan M.; Smith, Jan; MacCallum, Robert; Cascallar, Eduardo
Publication Year:
1995
Report Number:
RR-94-47
TOEFL-RR-49
Source:
Document Type:
Subject/Key Words:
English as a second language interrater reliability Spoken Proficiency English Assessment Kit (SPEAK) test reliability test validity

Abstract

The purpose of this project was to investigate issues of reliability and validity arising from the local administration and scoring of the Spoken Proficiency English Assessment Kit (SPEAK), the "off-the-shelf" version of the Test of Spoken English (TSE), and to assess the validity of the SPEAK test as a predictor of classroom English ability for international teaching assistants (ITAs). The reliability issues addressed included interrater consistency within and across institutions, score comparability across institutions and Educational Testing Service (ETS), and differential weightings by individual raters, institutions, and TSE raters of the three diagnostic language factors in assessing the overall comprehensibility of spoken English. SPEAK tests were administered to 119 international teaching assistants at three participating institutions, arranged in a counterblanaced design with 12 ETS-scored tests of the same form, and scored by raters at each institution. For the predictive validity phase of the study, the SPEAK scores of the 59 ITAs who held classroom teaching responsibilities were used as predictive variables in multiple regression analyses. Criterion variables were student ratings of the ITA's spoken language use in the classroom and other instructional contexts as well as ratings of the ITA on several dimensions of teaching effectiveness from the Student Instructional Report (SIR), a standardized instructor/course rating instrument. Results of the study indicate that the locally administered and scored SPEAK test, like TSE, is a reasonably reliable instrument for local screening and an appropriate measure of the general communicative proficiency in spoken English of ITAs in U.S. instructional settings.