skip to main content skip to footer

Test Reliability as a Function of the Number of Choices Per Item

Author(s):
Coward, Ann F.; Lord, Frederic M.
Publication Year:
1950
Report Number:
RB-50-47
Source:
ETS Research Bulletin
Document Type:
Report
Page Count:
14
Subject/Key Words:
Constructed Responses, Distractors (Tests), Multiple Choice Tests, Test Format, Test Length

Abstract

In order to choose the most efficient number of choices per item to use in a multiple-choice test, it is necessary to determine the relation between the test's reliability and the number of choices. Reliability is the correlation between two equivalent forms of a test. The tests are equivalent in that the means and variances of the two tests are equal and that the mean inter-item covariance is the same within each test, and between the two tests. Carroll and later Plumlee have derived a formula. Increasing the number of choices per item results theoretically in an increase in the reliability of the test; a decrease in number of choices reduces the reliability. Equations are provided to estimate the reliability of a revised test when empirical measures of reliability, standard deviation, and mean of the original test are not available. The greater the variance of item difficulty, the more the reliability of the revised test will deviate from that of the original. Several graphs of one of the equations are presented here. A change in the number of choices per item might make possible a change in the total number of items which could be included in a given time unit. The effect of such a change on test reliability could be estimated from the Spearman-Brown formula relating test length and reliability. A graph of this formula is provided. (SGK).

Read More