skip to main content skip to footer

The Effect of Difficulty and Chance Success on Item-Test Correlation and on Test Reliability

Author(s):
Plumlee, Lynnette B.
Publication Year:
1950
Report Number:
RB-50-54
Source:
ETS Research Bulletin
Document Type:
Report
Page Count:
47
Subject/Key Words:
Difficulty Level, Guessing (Tests), Item Analysis, Test Items, Test Reliability

Abstract

Equations are provided for predicting the effect of chance success, relative to item difficulty, on item-test correlation and on test reliability. A series of parallel mathematics achievement tests was set up in which the same items were used in both multiple-choice and answer-only form. The observed multiple-choice item difficulty (proportion correct), item-test correlation (biserial correlation), and test reliability (correlation between parallel forms) were compared with the observed answer-only statistics and the statistics predicted for multiple-choice from the observed answer-only statistics using the prediction equations. It was found that most differences between predicted and observed multiple-choice statistics were not significant. The existing differences in the item-test correlations and the test reliabilities seemed to be fairly consistently in the direction of the answer-only statistics, indicating that the prediction equations for these two statistics tended to over-correct in the practical multiple-choice test situation. This tendency was to be expected since common test construction practice, which was followed in setting up the multiple-choice items for the current study, is to reduce the chance factor by including among the options wrong answers which are popular in answer-only form. It was concluded, therefore, that the expected lowering of item difficulty, item-test correlation, and test reliability when answer options are supplied can be at least partially counteracted by careful item construction.

Read More