Development of Procedures for Resolving Irregularities in the Administration of the Listening Comprehension Section of the TOEFL® Test

Author(s):
Way, Walter D.; McKinley, Robert L.
Publication Year:
1991
Report Number:
RR-91-06
TOEFL-TR-03
Source:
Document Type:
Subject/Key Words:
Analysis of covariance Bayesian statistics listening comprehension statistical analysis test administration

Abstract

The purpose of this study was to develop and evaluate two procedures to be used in determining whether examinees in a given test center are affected by a testing irregularity on the Listening Comprehension section of the TOEFL® test (Test of English as a Foreign Language™). One approach employs analysis of covariance (ANCOVA) on Listening Comprehension (Section 1) means using scores on Structure and Written Expression (Section 2) and scores on Reading and Vocabulary (Section 3) as covariates. The second procedure entails a Bayesian approach that uses prior information collected about performance at the center in question. Analyses using these two procedures were carried out using both simulated data and data from actual testing irregularities. The results of this study support the following conclusions. First, both the ANCOVA and Bayesian procedures appear to provide useful information related to the effects of testing irregularities on Section 1 of the TOEFL test; therefore, both procedures should be incorporated into the operational procedures for resolving testing irregularities. Second, the two procedures will usually agree about the effects of testing irregularities, although differences between the procedures may occur in situations where the Bayesian procedure indicates an effect of one scaled score point and the ANCOVA procedure does not indicate a statistically significant difference between the irregularity and comparison groups at some specified level. Finally, results based on the two procedures should be interpreted with caution, particularly in situations where the two procedures produce discrepant results.