Mo Zhang on Evaluating Writing Processes

[background music starts]

On-screen: [ETS®. Measuring the Power of Learning®.]

On-screen: [Pulling to the Edge. Advancing assessment. Mo Zhang on Evaluating Writing Processes]

On-screen: [Mo Zhang, Research Scientist, NLP and Speech Group, ETS.]

[background music fades]

Speaker: Mo Zhang, Research Scientist, NLP and Speech Group, ETS — My name is Mo Zhang. I’m a Research Scientist in the natural language processing [on-screen: typographic treatment of the phrase "Natural Language Processing & Speech Research Group" appears with an illustrated caricature of seven people in a group setting] and speech group.

My colleagues and I have been conducting systematic research on writing processes [on-screen: the typographic treatment of the words "writing processes" replaces the phrase "Natural Language Processing & Speech Research Group" to the left of the illustrated caricature of seven people in a group setting] in the assessment context. When writing tasks [on-screen: the word "tasks" replaces the word "processes" so the typographic treatment reads "writing tasks"] are presented on the computer [on-screen: an illustrated caricature of a person sitting and typing on a computer appears] we can record the actions [on-screen: the words "actions and timing" appear on an illustrated computer paper print out that is flowing in the direction of the seven illustrated people in a group setting] and the associated timing information that students take during their writing of an essay. These logs [on-screen: a graphic grey spiral circles the end of the illustrated computer paper printout and a typographic treatment of the words "natural language processing" appear] can be further complemented by natural language processing techniques, such as recognizing [on-screen: a mustard colored graphic cylinder vessel appears and the type treatment of the words "natural language processing" disappear off the screen into the cylinder] the spelling errors [on-screen: the typographic treatment of the words "spelling errors" and "correction rate" appear in a circular graphic and then disappear from the screen into the cylinder] and the correction rate, and identifying the word types [on-screen: the typographic treatment of the words "word types" appear in a circular graphic and then disappear into the graphic cylinder as the typographic treatment of the words "linguistic complexity" appear in a circular graphic and then disappear from the screen into the cylinder] and their linguistic complexity. We can then use those features that characterize a student’s writing [on-screen: the typographic treatment of the words "features that characterize a student’s writing" appear on the graphic cylinder as the entire illustration moves to the background] to infer the underlying cognitive processes [on-screen: a graphic dotted arrow moves leftwards from the words "features that characterize a student’s writing" to the middle of the screen and the typographic treatment of the words "underlying cognitive processes" appears below the illustration of a person typing on a computer] a student engages during the composition. [On-screen: the previous illustration moves off the screen and a graphic infographic of the words "advanced," "proficient," "basic," "struggling," "planning," "editing" and "error correction" appear on top of graphic organic shapes above the typographic treatment of the words "which behaviors?"] We want to find the kinds of behaviors that positively [on-screen: three graphic arrows pointing upwards to an illustration of a hand-written document with the typographic treatment of the words "planning", "editing" and "error correction" paired individually to an arrow] relate to the final products. We also think that the process information gives us something new. It impacts how you [on-screen: a graphic olive squiggly line appears above the words "which behaviors" up to the illustration of the hand-written document] get there. [On-screen: the typographic treatment of the words "richer writing profile" appear above an illustrated caricature of an individual with a squiggly line from the individual to the illustration of the hand-written document on the right side of the screen] Essentially the writing process logs provide us with a much richer writing profile of students, which in turn allows us to better [on screen: typographic treatment of the words "represent" and "compare" appear, as does the illustration of another individual with a squiggly line from the second illustrated individual upwards to a second hand-written document] represent, compare and track students’ writing proficiency and [on-screen: two larger illustrated individuals appear to the right of the previous two with a squiggly line upwards towards two more hand-written documents] their growth over time.

[On-screen: the typographic treatment of the word "writing" appears under an illustrated caricature of an individual with a pencil in hand and an empty graphic of a thought bubble] Writing, as we know, is a complex mental process to [on-screen: the graphic of the thought bubble populates with an infographic of a light bulb and pencil and arrow] generate, express and refine ideas. Text production [on-screen: the typographic treatment of the word "text production" appears in the middle of the graphic thought bubble] involves at least four essential sub processes for writing. [On-screen:  the typographic treatment of the word "planning" appears] Planning, which refers to [on-screen: the typographic treatment of the words "generating ideas" appears as an off-shoot of the word "planning"] generating and organizing ideas [on-screen: the typographic treatment of the words "organizing ideas" appears as an off-shoot of the word "planning"].  [On-screen: the typographic treatment of the word "translating" appears to the right of the words "text production"] Translating, which is [on-screen: the typographic treatment of the words "turning ideas into words" appears as an off-shoot of the word "translating"] turning ideas into words. [On-screen: the typographic treatment of the word "transcribing" appears to the bottom right of the words "text production"] Transcribing, which is putting [on-screen: the typographic treatment of the words "writing of typing words" appear as an off-shoot of the word "transcribing"] words on a paper or typing words on the computer. Finally, there’s reviewing, [on-screen: the typographic treatment of the word "reviewing" appears with the words "reading & revising" appearing as an off-shoot] which is reading and revising the text, and the plan of the text.

[On-screen: the illustrated caricature of a person at a computer appears with the illustration of a group of seven people standing to the right of the computer behind a computer printout with the words "actions and timing" on it] We implement process feature collection in the CBAL® testing delivery software, [on-screen: The CBAL® Assessment appears typographically in the forefront of the screen with the descriptor "Cognitively Based Assessment of, for and as Learning" appearing beneath] so they can be used in other ETS operational testing programs. [On-screen: the illustrated caricature of the person begins typing and the computer screen populates with undiscernible text] We have addressed a series of research questions related to writing processes, and answers to these questions can help us understand [on-screen: the typographic treatment of the phrase "To what extent do process features relate to the quality of the final product?" appears on the screen above the illustrations] to what extent do process features relate to the quality of the final product.

[On-screen: the illustrated caricature of people shifts to the bottom left of the screen and an illustrated grey brick road appears in front of an illustrated mountainous setting with the word "Goals" appearing at the end of the road] There are several goals for our research; [on-screen: the typographic treatment of the phrase "generate new writing theory" appears in an arc above the grey brick road] to generate new writing theory, [on-screen: the previous illustrations disappear and the typographic treatment of "create computable features" appears with a graphic representation of water droplets falling into a cylinder-shaped container with the words "writing process features" written on it] to create computable features from that theory, [on-screen: the previous illustration disappears and the typographic treatment of the words "meaning & uses of features" appear with illustrated drawings of small tools] to study the meaning and potential uses of features for formative feedback in assessment context, [on-screen: the typographic treatment of the words "analytic & display capabilities" slide in from the right of the screen along with an illustrated magnifying glass hovering over a statistical graph] to develop analytical and displaying capabilities, to help our [on-screen: the previous illustration disappears and an illustrated caricature of an individual holding up an oversized graphic illustration of a ruler appears] assessment designers to design better tests, for [on-screen: a caricature of an illustrated researcher appears on screen with an illustration of a clipboard and a graphic check mark ticks in front of the ruler] researchers to gather evidence of validity of the assessments, and to develop a [on-screen: an illustrated visual of a computer screen with a reporting dashboard slides onto the screen from the right with typographic treatment of the words "reporting dashboard" above it] reporting dashboard that can really help teachers  improve their teaching of writing in classrooms.         

[On-screen: a caricature of an individual typing on computer screen appears with the words "writing process logs" on a graphic iteration of a computer printout and the word "FEEDBACK" appearing in large text] The writing process logs can give teachers and students feedback to improve writing practice. [On-screen: an illustrated caricature of a teacher sitting at a desk in front of a computer screen with information on it appears on screen] It allows the teachers to see how a student writes an essay and identifies gaps [on-screen: the typographic treatment of the words "error correction rate," "production rate," "time on task," "vocabulary sophistication," "evidence of editing" and "syntax variety" appear on screen individually] and difficulties in students’ writing process. A limited time on task could indicate a lack of motivation for which the teachers can intervene. A lack of editing behavior in combination with a piece of disjointed text could prompt teachers to focus more on editing strategies. [On-screen: the caricature of five illustrated people each appearing a little bit taller than the one to their left appears on screen with the typographic treatment of the words "high performers" angled upwards above their heads] Our research has found that higher performing students generally [on-screen: the typographic treatment of the phrases "spend more time on task," "more fluent at putting words on screen," "discard less of what they’ve written" and "show more evidence of editing" appear on screen under a visual of a graphic clock] spend more time on task, are more fluent at putting words onto the screen, they discard less of what they’ve written, and they showed more evidence of editing.

In a recent study, we found that [on-screen: the graphic symbols for male and female each appear on their own stack of paper; the male having less paper in its pile versus the female] in comparison to males, [on-screen: the typographic treatment of the sentence "Female students entered text more fluently, engaged in more editing behavior, and paused less frequently between long bursts of text production" appears] female students enter text more fluidly, they engage in more editing behavior, they spend less time on real planning,  meaning that female students pause less frequently in between long bursts of text production. Studies like this show us [on-screen: a caricature of five individuals appears on a path in front of an illustrated mountainous setting with the typographic treatment of the words "writing process analysis" above the illustrations] the value of writing process analysis in identifying and understanding population differences, as well as generating new hypotheses that will require further research. [On-screen: The ETS logo and tagline "Measuring the Power of Learning" appear at the top of the screen] ETS is one of the first to conduct such extensive and rigorous research in this line.    

[background music resumes]]

[On-screen: ETS®. Measuring the Power of Learning®. Writing Process Analysis.  Mo Zhang, Paul Deane, Randy Bennett, Peter van Rijn. Learn more about ETS research at www.ets.org/research. Copyright© 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service ETS.]

[On screen: Zang, M. & Deane, P. (2015). Process features in writing: Internal structure and incremental value over product features (Research Report No. RR-15-27). Princeton, NJ: Educational Testing Service.

Deane, P. & Zhang, M. (2015). Exploring the feasibility of using writing process features to assess text production skills (Research Report No. RR-15-26). Princeton, NJ: Educational Testing Service.

Zhang, M., Bennett, R. E., Deane, P., & van Rijn, P. (2016, July). Exploring gender differences in essay writing processes. Presentation at the Psychometric Society Conference. Asheville, NC.

Zhang, M., Zou, D., Wu, A. D., Deane, P., & Li, C. (2017). An investigation of writing processes employed in scenario-based assessment. In B. D. Zumbo & A. M. Hubley (Eds) Understanding and investigating response processes in validation research. New York: Springer].

[On-screen: Learn more about ETS research at www.ets.org/research]

[On-screen: Copyright© 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS).]

[music ends]

[End of Mo Zhang on Evaluating Writing Processes Video]

Video duration: 5:33