skip to main content skip to footer

A Preliminary Investigation of the Factors Related to the Design and Scoring of Video-Based Oral Communication Performance Tasks in Higher Education

Author(s):
Roohr, Katrina Crotts; Burkander, Kri; Mao, Liyang
Publication Year:
2018
Report Number:
RR-18-09
Source:
ETS Research Report
Document Type:
Report
Page Count:
23
Subject/Key Words:
Test Design, Test Scoring, Video Evaluation, Oral Communication, Higher Education, Interrater Reliability, Performance Assessment, Task Design, Large-Scale Assessment, Self Report, Scoring Rubric, Test Preparation

Abstract

Oral communication has been identified as an important skill by higher education institutions and by the workforce community. Despite its importance, minimal research has been conducted around the development of tasks to measure oral communication skills and behaviors. The purpose of this preliminary study is to evaluate the different factors related to the design and scoring of oral communication tasks. For this study, oral communication refers specifically to video‐based oral presentations using a webcam. The study administered 2 informative and 2 persuasive speech tasks of varying length (2 or 4 minutes) to 8 college students. Performance was scored holistically on overall performance, and on content and delivery dimensions by 2 raters. Results show that (a) interrater reliability estimates may be impacted by task type and length, (b) there is no variation in the three scoring criteria, (c) a minimum of 2 tasks is necessary to achieve satisfactory reliability on an oral communication measure, and (d) supporting materials may impact student performance. Results of this preliminary study provide guidance on how to further this foundational research. Future research and next steps are discussed.

Read More