skip to main content skip to footer

Assessing Mode Effects of At‐Home Testing Without a Randomized Trial

Author(s):
Kim, Sooyeon; Walker, Michael
Publication Year:
2021
Report Number:
RR-21-10
Source:
ETS Research Report
Document Type:
Report
Page Count:
21
Subject/Key Words:
Proctoring, COVID-19, Randomized Field Trial, Mode Effect, Online Assessment, Pseudo-Equivalent Groups, Equating, Delta Method, Test Center Services, Item Analysis

Abstract

In this investigation, we used real data to assess potential differential effects associated with taking a test in a test center (TC) versus testing at home using remote proctoring (RP).We used a pseudo-equivalent groups (PEG) approach to examine group equivalence at the item level and the total score level. If our assumption holds that the PEG approach removes between-group ability differences (as measured by the test) reasonably well, then a plausible explanation for any systematic differences in performance between TC and RP groups that remain after applying the PEG approach would be the operation of test mode effects. At the item level, we compared item difficulties estimated using the PEG approach (i.e., adjusting only for ability differences between groups) to those estimated via delta equating (i.e., adjusting for any systematic differences between groups). All tests used in this investigation showed small, nonsystematic differences, providing evidence of trivial effects associated with at-home testing. At the total score level, we linked the RP group scores to the TC group scores after adjusting for group differences using demographic covariates. We then compared the resulting RP group conversion to the original TC group conversion (the criterion in this study). The magnitude of differences between the RP conversion and the TC conversion was small, leading to the same pass/fail decision for most RP examinees. The present analyses seem to suggest little to no mode effects for the tests used in this investigation.

Read More