skip to main content skip to footer

Generalizability, Validity, and Examinee Perceptions of a Computer-Delivered Formulating-Hypotheses Test

Bennett, Randy Elliot; Rock, Donald A.
Publication Year:
Report Number:
ETS Research Report
Document Type:
Page Count:
Subject/Key Words:
Computer Assisted Testing, Formulating Hypotheses (Test), Generalization, Test Reliability, Test Takers, Test Validity


Formulating-Hypotheses (F-H) items present a situation and ask the examinee to generate as many explanations for it as possible. This study examined the generalizability, validity, and examinee perceptions of a computer-delivered version of the task. Eight F-H questions were administered to 192 graduate students. Half of the items restricted examinees to 7 words per explanation, and half allowed up to 15 words. Generalizability results showed high interrater agreement, with tests of between two and four items scored by one judge achieving coefficients in the .80s. As in studies of paper-and-pencil versions, validity analyses found that although F-H was highly reliable, it was only weakly related to GRE General Test Scores, differing from that test primarily in relating more strongly to a measure of ideational fluency. Versions of F-H based on different response limits tapped somewhat different abilities, with items employing the 15-word constraint appearing more useful for graduate assessment. These items added to conventional measures in explaining school performance and creative expres- sion. Finally, although the overwhelming majority of examinees found the F-H interface easy to use, some experienced difficulty, suggesting the possibility that computer familiarity constitutes a source of irrelevant variance in F-H scores.

Read More