skip to main content skip to footer

A Simulation‐Based Method for Finding the Optimal Number of Options for Multiple‐Choice Items on a Test

Author(s):
Guo, Hongwen; Zu, Jiyun; Kyllonen, Patrick C.
Publication Year:
2018
Report Number:
RR-18-22
Source:
ETS Research Report
Document Type:
Report
Page Count:
19
Subject/Key Words:
Multiple Choice Items, Simulation Methods, Test Reliability, Item Discrimination (Tests), Distractors (Tests), Multiple Choice Tests, Data Sets, Guessing (Tests), Test Length

Abstract

For a multiple‐choice test under development or redesign, it is important to choose the optimal number of options per item so that the test possesses the desired psychometric properties. On the basis of available data for a multiple‐choice assessment with 8 options, we evaluated the effects of changing the number of options on test properties (difficulty, reliability, and score comparability) using simulation. Using 2 criteria (low frequency and poor discrimination) to remove nonfunctioning options and 2 schemes (random and educated guessing) to model hypothetical response behavior for the removed options, we found that decreasing the number of options (from 8) created an easier test form but that a test form with reduced options could be more reliable if low‐discriminating options were removed and an educated guessing strategy were assumed. We present a rationale for the optimal number of options for this test being approximately 5, which would result in a shorter test while preserving its psychometric quality. Simulation methods discussed in this report could be applied to any test to compare the effects of changing the number of options.

Read More