GRE Analytical Reasoning Item Statistics Prediction Study

Author(s):
Boldt, Robert F.
Publication Year:
1998
Report Number:
RR-98-37
GREB-94-02P
Source:
Document Type:
Subject/Key Words:
Prediction studies statistical analysis GRE Analytical ability measure

Abstract

Chalifour and Powers (1989) noted that the ability to predict item statistics might be used to reduce the volume of item pretesting (Mislevy, Sheehan, & Wingersky, 1993). That study examined prediction of GRE analytical reasoning item statistics. Linear regression was used by these authors, but they surmised that non-linear techniques might provide somewhat better prediction of item statistics. Chalifour and Powers had amassed item statistics on a very large sample of analytical reasoning items. That sample was used in the present study. For the present study, predictions were generated using a type of neural net. This technique did indeed provide more accurate predictions of item difficulty and item discrimination in an estimation sample. However, when the functions developed in the estimation sample were cross-validated in a fresh sample, the advantages noted in the estimation sample disappeared. Only by including expert judgements of items difficulty with other predictors was the accuracy of prediction improved. The search used the validity of neural nets computed during operation of the algorithm to evaluate the efficacy of prediction. The genetic algorithm accomplished some improved prediction of item difficulties in the estimation, but no improvement with regard to discrimination. This examination used the validation sample. In sum, application of the complex and computer-intensive neural nets and genetic algorithms revealed no advantage over linear methods for predicting item difficulty and item discrimination statistics for GRE analytical reasoning items.

Read More