skip to main content skip to footer

A Proof-of-Concept Study on Scoring Oral Presentation Videos in Higher Education

Author(s):
Feng, Gary; Joe, Jilliam N.; Kitchen, Christopher; Mao, Liyang; Roohr, Katrina Crotts; Chen, Lei
Publication Year:
2019
Report Number:
RR-19-22
Source:
ETS Research Report
Document Type:
Report
Page Count:
28
Subject/Key Words:
Oral Communication, Human Scoring, Multimodal Assessment, Public Speaking, Videos, Transcripts, Vignettes, Speaking Assessments, Automatic Speech Recognition, Test Scoring, Higher Education

Abstract

This proof‐of‐concept study examined the feasibility of a new scoring procedure designed to reduce the time of scoring a video‐based public speaking assessment task. Instead of scoring the video in its entirety, the performance was evaluated based on content‐related (e.g., speech organization, word choice) and delivery‐related (e.g., vocal expression, nonverbal behaviors) dimensions. Content‐related dimensions were scored based on speech transcripts, while delivery dimensions were scored using a video thin‐slicing technique, where scores were assigned based on brief vignettes of a video rather than the complete performance. Initial feasibility data were collected from 4 novice raters scoring 10 video performances. Results indicated that for delivery dimensions, four 10 second thin slices yielded interrater consistency reliability somewhat similar to that of full‐video scoring, and additional slices resulted in only small improvements. For transcription‐based scoring, while raters were consistent, their scores had low correlations with criterion scores, which was likely due to differences in the modality (video vs. text). Video thin‐slicing appears to be a promising scoring technique for relevant constructs. Further testing of a combination of audio and transcript is recommended for scoring content‐related constructs.

Read More