skip to main content skip to footer

Development and Validation of the Written Communication Assessment of the HEIghten Outcomes Assessment Suite

Author(s):
Rios, Joseph A.; Sparks, Jesse R.; Zhang, Mo; Liu, Ou Lydia
Publication Year:
2017
Report Number:
RR-17-53
Source:
ETS Research Report
Document Type:
Report
Page Count:
18
Subject/Key Words:
Test Development, Test Validity, Written Communication, HEIghten, Writing Assessment, Student Learning Outcomes, Automated Scoring and Natural Language Processing, Item Analysis, Selected Response Items, Constructed Response Items, Dimensionality, Test Reliability, Motivation

Abstract

Proficiency with written communication (WC) is critical for success in college and careers. As a result, institutions face a growing challenge to accurately evaluate their students' writing skills to obtain data that can support demands of accreditation, accountability, or curricular improvement. Many current standardized measures, however, lack the ability to balance authenticity (i.e., requiring students to produce a sample of writing) with psychometric quality. To this end, we discuss the development of a newly developed measure, the WC assessment of the HEIghten outcomes assessment suite, and present pilot test results based on a sample of 985 test takers from 33 higher education institutions. Overall, we found that the measure includes well-functioning items (i.e., highly discriminating and lacking gender-differential item functioning), an essay task that can be reliably scored by combining human scores with scores provided by an automated algorithm, evidence to support reporting separate selected-response and essay scores to individuals and institutions, and adequate convergent validity evidence. Such results suggest that the HEIghten WC assessment demonstrates promise in providing institutions with a time- and cost-efficient measure of WC that may allow for actionable data to drive decision-making and improve teaching and student learning.

Read More