skip to main content skip to footer

Writing Processes in Short Written Responses to Questions Probing Prior Knowledge

Author(s):
Deane, Paul; O'Reilly, Tenaha; Chao, Szu-Fu; Dreier, Kelsey
Publication Year:
2018
Report Number:
RR-18-39
Source:
ETS Research Report
Document Type:
Report
Page Count:
32
Subject/Key Words:
Background Knowledge, Process Data, Keystroke Log, Writing Processes, Knowledge Telling, Multiple-Choice Items, Constructed-Response Items, Data Analysis, Short Answer Questions, Item Design, Test Scores

Abstract

The purpose of the report is to explore some of the mechanisms involved in the writing process. In particular, we examine students' process data (keystroke log analysis) to uncover how students approach a knowledge‐telling task using 2 different task types. In the first task, students were asked to list as many words as possible related to a particular topic (word listing). In a second task, students were asked to write to a specific prompt that was designed to elicit their background knowledge of a topic using connected text (knowledge elicitation). Using a matrix incomplete block design, 1,592 high school students completed the 2 writing tasks in addition to a multiple‐choice test of their background knowledge in 2 of 5 possible topics in the domain of U.S. history. An array of process data including students' typing and associated timing features was used to predict the writing scores on the 2 different types of tasks. The analyses revealed several distinct patterns that were associated with processing at the task knowledge productivity level, the editing effort level, and the keyboarding effort level. The robustness of the features was reflected in a set of hierarchal regressions that demonstrated that the process features were predictive of the writing score even when students' knowledge scores on the associated multiple‐choice test were considered. In sum, the results indicate that process data in the form of log file analysis are useful for both understanding the writing process and exploring potential differences between students with high and low knowledge.

Read More