skip to main content skip to footer

A Study of Confidence and Accuracy Using the Rasch Modeling Procedures IRT

Author(s):
Paek, Insu; Lee, Jihyun; Stankov, Lazar.; Wilson, Mark
Publication Year:
2008
Report Number:
RR-08-42
Source:
ETS Research Report
Document Type:
Report
Page Count:
25
Subject/Key Words:
Confidence, Rasch Model, Accuracy, Multidimensional Item Response Theory (MIRT), Item Response Theory (IRT)

Abstract

This study investigated the relationship between students’ actual performance (accuracy) and their subjective judgments of accuracy (confidence) on selected English language proficiency tests. The unidimensional and multidimensional IRT Rasch approaches were used to model the discrepancy between confidence and accuracy at the item and test level and to assess disattenuated strength of association between accuracy and confidence. The analysis results indicate a pattern of overconfidence bias (i.e., overestimation of success rate), which was related to item difficulty. In addition, the strength of association between accuracy and confidence dimension was relatively high: The confidence dimension explained 45% and 52% of the variability in the accuracy dimension for the two tests employed in this study.

Read More