skip to main content skip to footer

Properties of Three Statistics Used to Monitor the Quality of Constructed- Response Scoring Across Occasions QWK

Author(s):
McClellan, Catherine; Donoghue, John R.; Gladkova, Lydia
Publication Year:
2023
Report Number:
RM-23-09
Source:
ETS Research Memorandum
Document Type:
Report
Page Count:
27
Subject/Key Words:
Constructed-Response Scoring, Rescoring, Kappa Coefficient, Quadratic Weighted Kappa (QWK), Null Hypothesis Testing, Odds Ratio, Monitoring Scoring, Trend Scoring Method, Probability

Abstract

When constructed-response items are scored on separate occasions, one issue of interest is whether the scores assigned in the second context/time are equivalent to those assigned in the first. Research has demonstrated that the selection of papers at Occasion A (the rescore design) can have profound effects on the common measures of rescore, the percent exact agreement and the paired t-test. To be independent of the rescore design, cross-occasion inference must be based on conditional probabilities (given X subscore A is the original score for a response and X subscore B is the second score, the quantity gamma subscript n | m = P ((X subscript B equals n | X subscript A = m)). The appropriate null hypothesis is that the conditional probabilities of gamma subscript n | m on Occasion A are the same as the conditional probabilities at Occasion B of eta subscript n bar m for all values of n and m. This manuscript examines three other statistics—Cohen’s kappa, quadratically weighted kappa, and the odds-ratio—under the null hypothesis that the scoring is the same on Occasion B as it was on Occasion A of gamma equals eta. Strictly as a function of the rescore design, both kappa and quadratically weighted kappa yield expected values that are higher or lower than computed for the original within-occasion scoring. The odds ratio is shown to not be affected by the rescore design, but it is unclear how best to apply odds ratios for assessment of rater association in tables larger than two by two.

Read More