Test Validity by Design: A Framework for the TOEFL® Essentials™ Test
June 2, 2021
How can a test be both a valid and reliable measure of English-language proficiency while also providing test takers with a student-friendly, convenient test taking experience? By design. The recently launched TOEFL® Essentials™ test was designed specifically to provide the ideal combination – the quality institutions trust, and affordable access test takers want. Although offering valid and reliable information is not a new concept, the recently released test accomplishes this in a brief test-taking time, at an affordable price, using a format that is intended to be test-taker friendly and engaging. Along with the launch of the new test comes a 50-page publication, Design Framework for the TOEFL® Essentials™ Test 2021 (PDF), which displays the content of the test and its scoring procedures and presents a detailed research agenda supporting the use of the scores.
How the framework supports the validity of test scores
The test’s design framework is published to show institutions that they can trust the TOEFL Essentials test because of its innovative approach to test design. The design of the TOEFL Essentials test is the result of collaboration among ETS’s language learning researchers, content developers, psychometricians, and business-focused leaders within the TOEFL® program. It was also informed by input from extensive market research, including nearly 250 score users from institutions in the United States, Canada, and the U.K. and over 7,000 test takers around the world.
The new test includes tasks that were pilot tested with thousands of test takers around the world and cover two key dimensions of language proficiency: foundational aspects of language competence and task-based communicative language ability in academic and daily life contexts. By combining these key dimensions, the TOEFL Essentials test measures what test takers can do when using English, not just their knowledge of English. Unlike the constrained and inauthentic tasks found in other language tests administered from home, the tasks in the TOEFL Essentials test are designed to provide direct information about how successful test takers will be in the real world. For example, test takers will need to contribute to an academic-focused conversation or write to their professors and friends.
In addition to its approach to test design, TOEFL Essentials also boasts an innovative multistage adaptive test methodology. This methodology allows for efficient measurement of language proficiency, by administering the listening, reading, and writing sections in two parts to better match the individual language ability of the test takers. The first part contains test tasks of average difficulty, and responses to this part determine how difficult the tasks in the second part will be. Unlike at home tests that rely on adaptivity “on the fly,” the multistage adaptive methodology of the TOEFL Essentials test allows us to deliver robust communication tasks, assembled for each stage by expert assessment specialists, and targeted at the test taker’s proficiency level.
A legacy of research
The test’s design framework outlines numerous reasons why users can trust the test in addition to its validity evidence. With comprehensive security procedures, transparent score information, and the inclusion of a Personal Video Statement, these elements make the TOEFL Essentials test a trustworthy and test taker friendly test to demonstrate their English language abilities. For more than 50 years, the TOEFL Family of Assessments has been backed by high-quality, rigorous research, and the TOEFL Essentials test is no exception. Like all other tests in the TOEFL Family of Assessments, the research program for the TOEFL Essentials test drives all stages of test development, score interpretation and score use. The research program follows an argument-based approach to test validation, and it is presented in a series of questions which dictate the empirical evidence that should be collected. This research program is overseen by the TOEFL Committee of Examiners (COE), a panel of 11 experts from around the world, each of whom has achieved professional recognition in an academic field related to the teaching, learning, and testing of English as a second or foreign language.
To learn more about the TOEFL Essentials test, visit https://www.ets.org/toefl/score-users/essentials/about.html.
Spiros Papageorgiou is a managing senior research scientist at ETS.