skip to main content skip to footer

Agreement of Teachers on Evaluating Assessments of Learning Progressions in English Language Arts and Mathematics ELA

Author(s):
van Rijn, Peter; Graf, Edith Aurora; Arieli-Attali, Meirav; Song, Yi
Publication Year:
2018
Report Number:
RR-18-11
Source:
ETS Research Report
Document Type:
Report
Page Count:
27
Subject/Key Words:
Test Evaluation, Learning Progressions, English Language Arts (ELA), Mathematics, Summative Assessment, Teachers, Text Difficulty, Tasks, Human Raters, Agreement

Abstract

In this study, we explored the extent to which teachers agree on the ordering and separation of levels of two different learning progressions (LPs) in English language arts (ELA) and mathematics. In a panel meeting akin to a standard‐setting procedure, we asked teachers to link the items and responses of summative educational assessments to LP levels. We evaluated four types of agreement among teachers, with test developers, and with trained raters. Although the results were quite mixed, agreement on the LP levels of mathematics items was generally better than that for ELA tasks. The agreement on the LP levels of ELA sample responses was generally better than that for mathematics. Implications of the results are discussed.

Read More