skip to main content skip to footer

Scoring and Response Style on Written Simulation Problems

Author(s):
Alwood, James E.; Scheuneman, Janice Dowd
Publication Year:
1982
Report Number:
RR-82-02
Source:
ETS Research Report
Document Type:
Report
Page Count:
37
Subject/Key Words:
In-Basket Tests, Medical Examinations, Response Style (Tests), Scoring, Test Reliability

Abstract

Written Simulation Problems are a form of examination which offers an alternative to multiple choice testing for the evaluation of problem solving ability in fields such as medicine or other health professions. Although these problems have been used for assessment and evaluation for over a decade, relatively little attention has been paid to the psychometric properties of the scores derived from this type of examination. Of particular concern in this study are those scores derived for problem sections in which the examinee controls the number of options to be selected. Eight different procedures for scoring these sections were examined with respect to the impact of different types of error on the score, the susceptibility of the scoring procedure to response tendencies toward selecting many rather than few options, and the effects on the score of different examinees responding to different sections of the problem. Reliability of the scores was also determined. Results showed that the scores were generally adequate measures if the problems were balanced with regard to positive and negative options. With increasing imbalance, however, several of the scores became unsatisfactory, including some of the more commonly recommended. Suggestions for improved design and analysis of the problems were also made. (37pp.)

Read More