Video duration: 6:46
Test-based Accountability Systems: The Importance of Paying Attention to Consequences
On-screen: [ETS’s® 17th William H. Angoff Memorial Lecture. Test-based Accountability Systems: The Importance of Paying Attention to Consequences. Suzanne Lane, Ph.D. October 16th, 2019. ETS®]
Speaker: ETS® Suzanne Lane, Professor, Research Methodology Program, University of Pittsburgh
Hi, I’m Suzanne Lane. I’m a professor at the University of Pittsburgh in the Research Methodology Program, and I am honored to be the presenter at this year’s ETS® Angoff Memorial Lecture.
On-screen: [What are the negative consequences of test-based accountability systems? Why are they important? ETS®]
Some of the negative consequences of test-based accountability systems include the narrowing of instruction for the tested areas, such as in math and ELA, to focus only on those content and skills that are being assessed by the test. In addition, some of the negative consequences include the narrowing of the curriculum, where those tested subjects are being the focus, whereas the non-tested subjects, such as social studies, music, and others are not the focus of the curriculum.
It’s important to ensure that students are being tested across the breadth of the curriculum and across the breadth of the standards for those tested areas. It’s also important to ensure that we do not have adverse impacts for those students that these tests are supposed to serve. And those students typically are the students living in economically deprived areas, African American students, also Hispanic students, English language learners.
On-screen: [Since the intent of test-based accountability systems is to improve educational opportunities for all students, is it truly possible for educators, test developers and policymakers to develop a comprehensive, systemic evaluation of consequences? Why or why not? ETS®]
I do think it is possible to have a comprehensive evaluation of the consequences, including policymakers, educators, test developers, researchers in thinking about what are the most important consequences. In prioritizing the consequences, there are four factors we could look at. The first factor is the competing values by the different stakeholders. Certain stakeholders value some consequences, whether they be positive or unintended negative consequences, over others. So we need to consider those.
A second factor in focusing our efforts and the evaluation of consequences is prioritizing the fundamental positive consequences of these test-based accountability systems. Those are improving the instruction for all students, closing the achievement gaps, and ensuring that there are not negative consequences for those students who have been historically underserved.
A third factor to look at is examining the severity of those negative consequences, so identifying which negative consequences or potential negative consequences are most severe to students. And the fourth factor would be to evaluate the extent to which the testing system could cause adverse impact for students.
On-screen: [You talked about a conceptual framework for evaluating consequences of assessments. Why is such a framework important and what features should it include? ETS®]
The conceptual framework that I am proposing involves a number of factors. One of them is that we need to consider the stakeholders with regard to test-based accountability systems. They’re policymakers, educators, administrators, students, the community, business leaders, advocacy groups, so we need to have their views on what they value. Another factor is the values that go into specifying the consequences, whether they be the intended positive consequences or potentially the unintended negative consequences. Different stakeholders value different consequences over others.
A major part of it is the nature of the consequences and where they stem from. They can stem from test-based score interpretations and uses, and they can stem from using tests as a lever of educational change. Both of those areas really need to be a focus. Not only the way in which we interpret scores and use scores, but also the lever that the tests are playing, or the tool that the test is playing in terms of changing the educational system, making changes in instruction, increasing student knowledge and skills with respect to the content standards.
Other factors to consider is the differential impact. This differential impact of these testing systems can result in negative consequences for different subgroups of the population. We also have to think about contextual features, such as the socioeconomic status of groups, the resources provided for different schools for improving education, and other factors.
On-screen: [What do you see for the future of test-based accountability systems? What should we do to minimize negative consequences? ETS®]
The future of test-based accountability systems is going to be perhaps lessening the stakes associated with these particular systems, and then developing assessments that really model what is valued in instruction. An example is the pilot program under ESSA that New Hampshire is working with. They’re working with a number of districts to develop performance-based assessments, some of them being district-developed, others being common across the districts. These performance-based assessments hopefully will model what is valued in instruction and be reflective of the culture that the students are growing up in.
In terms of minimizing the negative effects, I think lessening some of the stakes associated with the tests used in these accountability systems, and also to include assessments like this PACE program in New Hampshire that really reflect what is valued in instruction.
On-screen: [ETS® Copyright © 2019 by Educational Testing Service. All rights reserved. ETS® and the ETS® logo are registered trademarks of Educational Testing Service (ETS®).]
End of Video: ETS®. Test-based Accountability Systems: The Importance of Paying Attention to Consequences.