angle-up angle-right angle-down angle-left close user menu open menu closed search globe bars phone store

Using Large-Scale Assessments to Analyze Education, Skills and Economic Opportunity

Focus on R&D

Issue 14

July 2019

By: Hans Sandberg

For a long time, education was seen as a sure path to success, but changes over the past 40 years have raised questions about this belief. Researchers at ETS have explored these issues, building on their expertise in large-scale assessments, such as the National Assessment of Educational Progress (NAEP), the Programme for International Student Assessment (PISA) and the Programme for the International Assessment of Adult Competencies (PIAAC). Focus on ETS R&D asked Irwin Kirsch and Anita Sands, at the ETS Center for Research on Human Capital and Education, about the impact of education and skills on equity and economic opportunity.

Why is ETS studying the impact of and distribution of skills?

Photo of Irwin Kirsch Irwin Kirsch: It used to be that all you needed to land a good job was a high school diploma or its equivalent, but today you need both some level of postsecondary education or credential and the right skill set. Behind this change lies several decades of globalization, technological advances and changes in public policy. Today's employers look for better educated individuals, who have mastered a broad set of skills, along with the ability and initiative to learn on their own.

Photo of Anita Sands Anita Sands: Our research into the role skills play in the life outcomes of students and adults is in line with ETS's mission, which is to advance quality and equity in education for all learners. We have leveraged ETS's work on large-scale international assessments to produce a series of policy reports that inform the public about the acquisition, outcomes and inequalities of human capital. Essential literacy and numeracy skills, as well 21st century skills such as collaboration, critical thinking and problem-solving, are key components here. With these research reports, we have demonstrated the importance of understanding the return on these essential skills, as well as what factors determine who gets to develop them. It is against this background that we have created a framework for understanding and addressing the skills challenge. Back in 2016, we argued, in Choosing Our Future: A Story of Opportunity in America, that America is in the midst of a crisis of opportunity — one that is tightly linked to both the growing importance of, and inequitable distribution of key skills across our society. We believe that it takes a sustained, systematic approach focused on policies, practices and interventions to address this crisis. This is also why we have produced these research reports.

Could broader access to higher education solve the crisis of opportunity?

Photo of Irwin Kirsch Irwin Kirsch: The proportion of high school students who graduate on time is higher than ever before, and even those who don't, will in most cases earn a high school degree or certificate by their early 20s. Two-thirds of high school graduates continue to college. This looks reassuring for the future, but the image darkens if you zoom in on the essential skills that students and adults in the U.S. need. When we analyzed data from large-scale international assessments, we found that many with secondary and postsecondary degrees and certificates failed to demonstrate essential literacy and numeracy skills.

In fact, our Center recently published two research reports on literacy and numeracy skill proficiencies of millennials¹ in America. The reports showed that while this generation has achieved a higher level of education than any previous generation in American history, many still lack critical skills. No less than 60 percent of the millennials performed below what many experts agree is a minimum standard for numeracy, and nearly half performed below this minimum in literacy.

Photo of Anita Sands Anita Sands: There has been so much focus on credentials in the public debate, but a degree can create a false sense of achievement if not matched by adequate skills. This raises questions about public policies that single-mindedly push educational credentials, especially if we don't look at how the skills associated with those credentials are distributed.² I'm not saying that educational attainment is unimportant, but it takes more than a certificate or a diploma to be successful in the labor market. Literacy and numeracy skills are increasingly important for all aspects of life today, from navigating health care to dealing with the deluge of information and news we confront each day.

Paul Harrington, who is a labor economist at the Center for Labor Markets and Policy at Drexel University, and a coauthor of several studies published through our Center, has pointed out that the promise of education is diminished if we allow students to attain degrees and diplomas without adequate levels of literacy skills.

How is ETS using its experience of large-scale assessments to understand the relation between education, opportunity and skills?

Photo of Irwin Kirsch Irwin Kirsch: It is said that technology changes everything, and we've already seen the loss of millions of jobs in the U.S. and elsewhere. Add to that the question about the future impact of artificial intelligence and robotics on the labor markets and everyday life. Many countries are responding to these scenarios by investing in human capital. We have also seen a growing interest in large-scale national and international assessments of both students in-school and adult populations.

Our work in large-scale assessments began in 1983, when ETS proposed a new design and methodology for NAEP, which is also known as the Nation's Report Card. NAEP assessments provide valuable knowledge about the skills of America's fourth-, eighth- and 12th-grade student populations. ETS also plays an important role in major international large-scale assessments, such as the Organisation for Economic Co-operation and Development's (OECD) PISA, which looks at the reading, mathematics and science skills of 15-year-old students around the world, and PIAAC.

PIAAC provides an important complement to student assessments as it not only looks at cognitive skills of adults aged 16–65, but gathers information about family background, educational attainment, work and training history, skills use and income. Data from PIAAC provides a rich source of information that shows how important skills are distributed within and among countries. It enables researchers to understand relations between skills and individuals' participation in education, training and labor force. PIAAC also looks at social outcomes such as well-being and trust.

Photo of Anita Sands Anita Sands: These large-scale assessments look at different age groups and broad sets of skills, while our Center's work focuses on links between skills and educational, social and economic outcomes. We have, for example, shown that people with essential skills tend to have increased levels of trust in others and in institutions, as well as higher levels of civic engagement, health and lifelong learning. Seen in this perspective, I find it deeply troubling that nearly half of America's millennials — around 36 million — have low literacy skills, and judging from NAEP data, so do many in the coming generations.

How can this research help America face these challenges?

Photo of Irwin Kirsch Irwin Kirsch: We at the Center for Human Capital hope that our research can help the public and our policymakers understand the importance of essential skills for society. The series of recent reports we have published show that education and skills are key factors associated with opportunity in our country. This insight must, in my view, play a central role for any inclusive pathway toward the future. The fact that large numbers of Americans lack essential literacy and numeracy skills is a major stumbling block on this path. The challenge we as Americans now face is whether we will allow our nation to continue growing apart with respect to these factors or whether we will invest in efforts to help us grow together. I believe the answer is clear and that the time to act is now!

1 Sands, A., & Goodman, M. (2018). Too Big to Fail: Millennials on the Margins. ETS Center for Research on Human Capital and Education; Goodman, M., Sands, A., & Coley, R. (2015). America's Skills Challenge: Millennials and the Future, ETS Center for Research on Human Capital and Education.

2 Li, T., von Davier, M., Hancock, G., & Kirsch, I. (2016). The Prediction of Labor Force Status: Implications from the International Adult Skills Assessment, ETS Research Report, ETS RR-16-11.

Find out more about the impact education and skills have on equity and economic opportunity.

Irwin Kirsch holds the Ralph W. Tyler Chair in Large-Scale Assessment at ETS and leads its Center for Research on Human Capital and Education. Anita Sands is a Lead Policy Research Analyst for the Center.