Development of a Scale for Assessing the Level of Computer Familiarity of TOEFL Examinees

Author(s):
Eignor, Daniel; Taylor, Carol; Kirsch, Irwin; Jamieson, Joan
Publication Year:
1998
Report Number:
RR-98-07
TOEFL-RR-60
Source:
Document Type:
Subject/Key Words:
Computer literacy questionnaires measures individuals test bias computer-assisted testing factor analysis language proficiency English as a second language

Abstract

The increasing use of computer-based testing raises concerns about equity and bias. Specifically, many in the field of language testing are concerned that the introduction of a computer-based TOEFL test in 1998 may introduce bias by confounding the measurement of English-language proficiency with level of computer familiarity. As the first step in a series of research efforts to address this question, a 23-item questionnaire focusing on individuals' access to, attitude toward, and experience with using computers as well as related technology was developed and administered to TOEFL test takers in April and May of 1996 as part of a TOEFL test administration. Results of this survey administration are documented in a paper by Kirsch, Jamieson, Taylor, and Eignor (1998). This paper has two purposes. The first is to explain decisions as to which questionnaire items would subsequently be used together as a score to separate TOEFL examinees into levels of computer familiarity. The second purpose is to provide details on the procedures used to assess the underlying factor structure of the complete questionnaire. Factor analysis procedures (specifically, a principal factors approach with both varimax orthogonal and promax oblique rotations) were used for both purposes. Eleven items, which loaded heavily on the first factor of a two-factor common factor solution with promax oblique rotation, were selected to create a computer familiarity score. This score was then used to classify TOEFL examinees into one of three familiarity groups: low, moderate, and high. A four-factor common factor solution with promax oblique rotation was seen as providing the best accounting of the underlying factor structure of the complete questionnaire.

Read More