Validating the Revised TSE® Against a Criterion of Communicative Success
- Powers, Donald E.; Shedl, Mary A.; Wilson-Leung, Susan; Butler, Frances A.
- Publication Year:
- Report Number:
- RR-99-05, TOEFL-RR-63
- Document Type:
- Subject/Key Words:
- Construct validity English as a second language interrater reliability listening comprehension prediction teaching assistants undergraduate students
A communicative competence orientation was adopted to study the validity of test score inferences derived from the revised TSE® (Test of Spoken English™). To implement the approach, a sample of undergraduate students primarily native speakers of English, provided a variety of reactions to, and judgments of, the test responses of a sample of TSE examinees. The TSE scores of these examinees, previously determined by official TSE raters, spanned the full range of TSE score levels. Undergraduate students were selected as "evaluators" because they, more than most other groups, are likely to interact with TSE examinees, many of whom become teaching assistants. Student evaluations were captured by devising and administering a secondary listening test (SLT) to assess students' understanding of TSE examinees' speech, as represented by their tape responses to tasks on the test of spoken English. The objective was to determine the degree to which official TSE scores are predictive of listeners' ability to understand the messages conveyed by TSE examinees. Analysis revealed a strong association between TSE score levels and judgments, reactions, and understanding of listeners. This finding applied to all TSE tasks and to nearly all of the several kinds of different evaluations made by listeners. Along with other information, the evidence gathered here should help the TSE program meet professional standards for test validation. The procedures may also prove useful in the future test development efforts as a way of determining the difficulty of speaking tasks (and possibly writing tasks).