skip to main content skip to footer

Using a Neural Net to Predict Item Difficulty

Author(s):
Boldt, Robert F.; Freedle, Roy O.
Publication Year:
1996
Report Number:
RR-96-31
Source:
ETS Research Report
Document Type:
Report
Page Count:
26
Subject/Key Words:
Algorithms, Difficulty Level, Item Analysis, Neural Net, Prediction

Abstract

Several studies that attempted to predict item difficulty using linear weighted composites have been done at ETS and elsewhere. The variables in the composites were coded item characteristics. The validity of these predictions was evaluated using multiple correlations between predicted and actual item difficulties. Results of these studies indicate that the validities of the linear predictions were not zero, but neither were they high. Also, success varied with the particular prediction task attempted, with validities varying from .45 to .77. The figures were promising, but more precision would be helpful for item design and test specification. The present study sought to improve on the validity of the linear predictions by using a "neural net," a technique that is widely used for nonlinear prediction and is borrowed from artificial intelligence applications. Predictions made using the neural net might be more accurate than those made using linear prediction for two reasons: (a) the latter is a special case of neural net prediction, one in which the variables do not interact, and (b) it is thought that item characteristics do interact to produce item difficulty. Unfortunately, it was found that using the neural net did not lead to substantial improvement in prediction when using the same characteristics as arguments that had proven superior for linear prediction.

Read More