skip to main content skip to footer

Developing a Machine-Supported Coding System for Constructed-Response Items in PISA LSA PISA CBA

Author(s):
Yamamoto, Kentaro; He, Qiwei; Shin, Hyo Jeong; von Davier, Matthias
Publication Year:
2017
Report Number:
RR-17-47
Source:
ETS Research Report
Document Type:
Report
Page Count:
17
Subject/Key Words:
Coding, Constructed-Response Items, Automated Scoring and Natural Language Processing, Large-Scale Assessments (LSA), Programme for International Student Assessment (PISA), Computer Based Assessment (CBA)

Abstract

Approximately a third of the Programme for International Student Assessment (PISA) items in the core domains (math, reading, and science) are constructed-response items and require human coding (scoring). This process is time-consuming, expensive, and prone to error as often (a) humans code inconsistently, and (b) coding reliability in international large-scale assessments varies across items and countries. The shift in PISA 2015 from paper- to computer-based assessment digitized all responses and associated coding, thus providing opportunities to introduce technology and analytical methods to improve data processing and analyses in future cycles. In this computer-based setting, this research explains the framework and approach for improving the accuracy and efficiency of the coding process in constructed-response items for future PISA cycles. In the pilot study associated with this research, we demonstrate how the proposed machine-supported coding system can be utilized using sample reading items from the PISA 2015 cycle.

Read More