skip to main content skip to footer

The Impact of Aberrant Responses and Detection in Forced‐Choice Noncognitive Assessment MUPP

Author(s):
Kim, Sooyeon; Moses, Tim P.
Publication Year:
2018
Report Number:
RR-18-32
Source:
ETS Research Report
Document Type:
Report
Page Count:
17
Subject/Key Words:
Noncognitive Assessment, Forced Choice Technique, Multi-Unidimensional Pairwise Preference (MUPP), Aberrant Responses, Person-Fit Analysis, Item Formats, Accuracy, Markov Processes

Abstract

The purpose of this study is to assess the impact of aberrant responses on the estimation accuracy in forced‐choice format assessments. To that end, a wide range of aberrant response behaviors (e.g., fake, random, or mechanical responses) affecting upward of 20%–30% of the responses was manipulated under the multi‐unidimensional pairwise preference (MUPP) model with forced‐choice format. Under each aberrance condition, we also computed the lz statistic (a common person‐fit index) to determine how well this index can accurately detect aberrant responses without erroneously misidentifying honest examinees. Some aberrant behaviors (e.g., limited responses) were more problematic than others (e.g., omitting) in terms of true trait level recovery. The lz statistic associated with each dimension of the MUPP form led to higher detection rates than did the overall lz statistic. When the dimensional lz approach was used, however, many honest examinees were misclassified as outliers. The detection rates of lz differed as a function of aberrant response types.

Read More