The Relationship Between Computer Familiarity and Performance on Computer-Based TOEFL Test Tasks

Taylor, Carol; Jamieson, Joan; Eignor, Daniel; Kirsch, Irwin
Publication Year:
Report Number:
Document Type:
Subject/Key Words:
Computer-assisted testing language proficiency test bias analysis of covariance computer literacy paper and pencil tests


The increasing use of computer-based testing raises concerns about equity and bias. Specifically, many in the field of language testing are concerned that the introduction of a computer-based TOEFL test in 1998 will confound language proficiency with computer proficiency and thus bring construct-irrelevant variance to the measurement of examinees' English-language abilities. In a Phase I study (Kirsch, Jamieson, Taylor, & Eignor, 1998), TOEFL examinees were surveyed regarding their computer familiarity and classified into one of three computer familiarity groups: low, moderate, and high. In this study, Phase II, more than 1,100 "low-computer-familiar" and "high-computer-familiar" examinees from 12 international sites were identified from the Phase I survey and administered a computer tutorial and a set of 60 computer-based TOEFL test items. The relationship between level of computer familiarity and performance on the computer-based items was then examined. The examinees in Phase II were largely representative of those in Phase I, who were representative of the general TOEFL test-taking population. The effect of computer familiarity after adjustments for language ability was examined by performing a series of analyses of covariance (ANCOVAs), using TOEFL paper-and-pencil test score as the covariate. These analyses were followed by a series of ANCOVAs involving the computer familiarity variable and a number of other variables: gender, reason for taking the TOEFL test, times the TOEFL test had been taken, and location where the TOEFL test was taken. After controlling for language ability, the researchers found no meaningful relationship between level of computer familiarity and level of performance on computerized language tasks among TOEFL examinees who had completed the computer tutorial.

Read More


Find a Publication

Advanced Search

Closing the Achievement Gap

Closing the Achievement GapLearn more about ETS's commitment to closing the achievement gap through rigorous research, thought-provoking forums and more.