Measuring English Language Proficiency across Subgroups: Using Score Equity Assessment to Evaluate Test Fairness
- Yoo, Hanwook; Manna, Venessa F.; Monfils, Lora; Oh, Hyeon-Joo
- Publication Year:
- Language Testing, v36 n2 p289-309, Apr 2019
- Document Type:
- Page Count:
- Subject/Key Words:
- Background English Language Proficiency Population Invariance Score Equity Assessment (SEA) Second Language Learning Subgroups Test Fairness TOEIC Listening and Reading Tests
SUMMARY:English language proficiency assessments are designed for a targeted test population that may include test takers from diverse demographic, sociocultural, and educational backgrounds. The test is assumed to be fair and the scores earned by different subgroups of test takers have the same meaning. One way of evaluating the test fairness is to produce a linked test for each subgroup and compare the test score results of the linked test with the test scores of the original test they took.
Using a score equity assessment (SEA) approach, this study investigated the consistency of test scores across the diverse group of Japanese test takers who took the TOEIC test. Overall, the results for the TOEIC Listening and Reading test indicates that test takers who belong to different subgroups and who have the same score on one test have the same expected test score on the linked test.
The results of the study suggest that the SEA analysis using various test-taker background characteristics is a useful approach for ensuring the fairness of reported test scores and for understanding the impact of background characteristics on the test performance of assessments. This is of particular importance in the context of English language proficiency assessments, as the testing population tends to be widely divers and its composition is likely to change over time given the increasing use of English as a medium of instruction and as a means of communication among nonnative speakers of English globally. As a result, the outcome of this study supports the fairness of the TOEIC test scores and allow score users to make an equitable decisions based on the TOEIC scores.
 Score equity assessment (SEA) – A statistical approach to evaluate whether reported test scores on different forms that are supposed to be used interchangeably are in fact interchangeable across different subgroups such as male and female. The invariance of linking functions across subgroups assesses the degree of interchangeability of scores.