skip to main content skip to footer

The Evidence Trace File: A Data Structure for Virtual Performance Assessments Informed by Data Analytics and Evidence‐Centered Design EPCAL VPA ECD GBA DBA

Author(s):
Hao, Jiangang; Mislevy, Robert J.
Publication Year:
2018
Report Number:
RR-18-28
Source:
ETS Research Report
Document Type:
Report
Page Count:
18
Subject/Key Words:
Evidence-Centered Design (ECD), Game-Based Assessment (GBA), Virtual Performance Assessments (VPA), Log Files, Educational Testing Service Platform for Collaborative Assessment and Learning (EPCAL), Evidence Trace Files (ETF), SimCityEDU, HYDRIVE, Digitally Based Assessments (DBA), Data Analytics

Abstract

Virtual performance assessments (VPAs), such as game‐ and simulation‐based assessments, provide promising ways for assessing complex and integrated skills. However, the high cost, long development cycle, and complex scoring process significantly hinder the adoption of VPAs, particularly in large‐scale assessments with tight deadlines and limited budgets. One significant impediment is the lack of consensus about how to capture and store evidence‐bearing process data from each VPA session into a format that facilitates quality checks, scoring, and analysis. In current practice, data from VPAs are generally stored in logs, either as standalone files or as a set of entries in a database. Programmers have long used logs to save process information about software systems for monitoring and debugging the software system itself. These purposes are fundamentally different from the purpose of assessment, which aims to capture evidence to support the measurement of a test taker's proficiency with respect to targeted constructs. Collapsing information that might be needed for the two purposes into a single log file can lead to confusion, even confrontation, among VPA designers, software developers, and scoring experts. To mitigate the problem, in this paper we introduce the notion of a structured evidence trace file (ETF) as distinguished from the familiar but idiosyncratic term, log file. The ETF is intended to express captured telemetry data in a way that meets the needs from the assessment perspective. Its specification as a standalone deliverable allows VPA designers and scoring experts to cleanly define and assure the quality of the information they need from VPA sessions in forms they find most useful for assessment purposes without adding additional complexity or structures for the information that software developers need for debugging purposes. In turn, software developers will be clearly informed about what information is expected from the assessment perspective and can manage the data as they choose as long as they can deliver the specified ETF. In addition to its application in VPAs, the ETF structure can also be employed more generally with digitally based assessments (DBAs) in which response‐process data are captured and used for assessment purposes, such as timing information, response changes, and tool usage in multiple‐choice tests. As a known data structure, it can sometimes also be employed for additional purposes in VPA analysis, such as transmitting information across phases of multiple‐stage analysis.

Read More