Human Capital and International Large-Scale Assessment
As societies become increasingly advanced, adults are expected to use information in more complex ways and to maintain and enhance their skills in literacy, numeracy, science and problem-solving in technology-rich environments throughout their lifetimes. Such skills are critical not only for personal achievement, but also for positive social, educational and economic outcomes around the world.
ETS’s research in this area is related to international large-scale comparative surveys of adults and youth of school age. The focus is on core skills such as reading, math/numeracy and science literacy. Other areas addressed in these surveys include innovative constructs such as financial literacy, health literacy and problem solving in technology-rich environments.
ETS’s research in human capital and international large-scale assessment takes place both in the United States and around the world. Our large-scale international comparative surveys are designed to meet policy needs reflecting a growing interest in skills as human capital. These are some examples:
OECD Survey of Adult Skills
ETS manages this major assessment for the Organisation for Economic Co-operation and Development (OECD) as part of the Programme for the International Assessment of Adult Competencies (PIAAC). This is the first large-scale, computer-delivered comparative assessment of adults in the world. It provides policymakers with a profile of adults’ knowledge, skills and competencies. PIAAC expands upon earlier literacy assessments, and it is the first multistage adaptive test used in large-scale comparative assessment.
We also work on two other projects related to the PIAAC program:
- Education and Skills Online — A new computer-delivered assessment linked to PIAAC. It provides individual results for persons who want to demonstrate readiness skills, and allows institutions, organizations and local governments to assess a population for training and research purposes. Scheduled to be available in early 2015.
- The World Bank’s Skills Toward Employment and Productivity (STEP) — This program, which is part of PIAAC, measures reading literacy and component skills. It seeks to help countries assess how different skill sets affect labor-market opportunities, and provides information on employability and marketability.
In November 2013, ETS organized the PIAAC Invitational Research Conference, which highlighted the importance of the PIAAC results for understanding the role of skills in a global context. Watch the conference’s opening video.
Programme for International Student Assessment (PISA)
ETS leads this global evaluation of education systems. PISA is a multinational survey of 15-year-old students, currently administered in some 70 countries. The test, which has transitioned from paper to computers, assesses what students know and their ability to apply their knowledge in real-life scenarios at an age where they begin to fully participate in society.
TIMSS and PIRLS
ETS acts as consultant to Boston College on two international assessments: Trends in International Mathematics and Science Study, as well as the Progress in International Reading Literacy Study.
IEA-ETS Research Institute (IERI)
ETS collaborates with colleagues from around the world on international large-scale assessments. One such example is the IEA-ETS Research Institute (IERI) — a partnership between ETS and the International Association for the Evaluation of Educational Achievement (IEA). IERI publishes an international journal and runs workshops on methodology using existing large-scale assessment databases. Both independently and as part of its involvement in IERI, ETS’s researchers also produce research reports, books and book chapters related to human capital and international large-scale assessment (see list below).
Here is a selection of significant publications related to human capital and international large-scale assessment:
America's Skills Challenge: Millennials and the Future
M. Goodman, A. Sands & R. Coley
The report uses data from the PIAAC survey to compare the United States to 21 other member countries in the OECD, focusing on "millennials," i.e., young adults born after 1980 who were 16–34 years of age at the time of the assessment. On average, America's millennials demonstrated weak skills in literacy, numeracy and problem-solving in technology-rich environments compared to their international peers. View citation record
Handbook of International Large-Scale Assessment: Background, Technical Issues, and Methods of Data Analysis
Editors: L. Rutkowski, M. von Davier, & D. Rutkowski
Publisher: CRC Press
This book highlights the importance of international large-scale assessment data to policy and research. It reviews methodological aspects and features of the studies based on operational considerations, analytics and reporting. Chapter authors describe methods of interest to readers who have a good grounding in quantitative methods, but who are not necessarily quantitative methodologists. View citation record
The Role of International Large-Scale Assessments: Perspectives from Technology, Economy, and Educational Research
Editors: M. von Davier, E. Gonzalez, I. Kirsch, & K. Yamamoto
This ETS-edited volume compiles interdisciplinary papers on research using international large-scale assessment data. It includes perspectives from economics, policy, education, civics education and educational technology. Chapter authors discuss the challenges that come with the increasing interest and the broadening scope of international large-scale assessments. View publisher's page
Technical Report of the Survey of Adult Skills (PIAAC)
Publisher: Organisation for Economic Co-operation and Development (OECD)
This report, with contributions from ETS and other members of the consortium involved with PIAAC, describes in detail the design, development and data analysis used for the first computer-delivered large-scale assessment of adult literacy. View full report
Fault Lines in Our Democracy: Civic Knowledge, Voting Behavior, and Civic Engagement in the United States
R. Coley & A. Sum
ETS Policy Information Center Report
The report examines gaps in educational attainment and income among different segments of the U.S. population with regards to civic knowledge, voting behavior and civic engagement. View citation record
An Experimental Study of the Effects of Monetary Incentives on Performance on the 12th-Grade NAEP Reading Assessment
H. Braun, I. Kirsch, & K. Yamamoto
Teachers College Record, Vol. 113, No. 11, pp. 2309–2344
This article reports on a study of monetary incentives for 12th-grade students taking an assessment designed to replicate the NAEP reading test. The study concluded that NAEP may both underestimate the reading abilities of students enrolled in 12th grade and yield biased estimates of certain achievement gaps. View publisher's page
Measuring Growth in a Longitudinal Large Scale Assessment with a General Latent Variable Model
M. von Davier, X. Xu, & C. H. Carstensen
Psychometrika, Vol. 76, No. 2, pp. 318–336
This article presents approaches for use with longitudinal large-scale assessment data, i.e., data that were repeatedly collected on the same sample of students or adults. The article describes models that made it possible to examine whether the Matthew-effect (those who have more, gain more) can be identified in a national extension of the PISA assessment. View citation record
Investigation of Model Fit and Score Scale Comparability in International Assessments
M. E. Oliveri & M. von Davier
Psychological Test and Assessment Modeling, Vol. 53, No. 3, pp. 315–333
This article presents evidence on scale stability and the need to account for national variations in how some tasks in international assessments are understood. While rigorous translation, verification and assessments of cultural appropriateness reduce the threat of cross-country differences in how tasks are understood, it is always possible to detect deviations from the international standard on a limited number of tasks. This article presents an approach of how such deviation can be remedied. View article
Relationships Among Reading Skills of Adults with Low Literacy
J. P. Sabatini, Y. Sawaki, J. R. Shore, & H. S. Scarborough
Journal of Learning Disabilities, Vol. 43, No. 2, pp. 122–138
In this study, researchers used a method known as confirmatory factor analysis to examine the relationship between literacy and several component reading skills (e.g., word recognition, fluency, language comprehension and vocabulary skills) in adults. View publisher's page
Basic Reading Skills and the Literacy of America's Least Literate Adults: Results from the 2003 National Assessment of Adult Literacy (NAAL) Supplemental Studies
J. Baer, M. Kutner, & J. Sabatini
National Center For Education Statistics Report No. NCES 2009481
The 2003 NAAL assessed the English literacy skills of U.S. adults. This report focuses on results from the supplemental basic reading skills measures given to all adults in the survey, as well as a special battery of assessment given to the least literate subpopulation. View citation record
What are Plausible Values and Why Are They Useful?
M. von Davier, E. Gonzalez, & R. J. Mislevy
IERI Monograph Series: Issues and Methodologies in Large Scale Assessments, Vol. 2.
This report describes one of the fundamental tools developed to enable secondary analysis of large-scale assessment data through plausible values, a form of imputations that incorporates all available information into proficiency values. The report provides a less technical and more practical overview of the use of this methodology and contracts appropriate analysis with shortcuts that lead to biased analysis results. View the article
Preparing for an Epidemic of Limited Health Literacy: Weathering the Perfect Storm
R. M. Parker, M. S. Wolf, & I. Kirsch
Journal of General Internal Medicine, Vol. 23, No. 8, pp. 1273–1276
Research has demonstrated strong links between literacy and health. ETS's recent report America's Perfect Storm considers the great risks the United States faces as a result of declining adult literacy, shifting demographics and a changing economy. It is essential to understand how these changes will impact health care. View publisher's page
America's Perfect Storm: Three Forces Changing Our Nation's Future
I. Kirsch, H. Braun, K. Yamamoto, & A. Sum
ETS Policy Information Report
This report looks at the convergence of three powerful sociological and economical forces that are changing the future of the United States: substantial disparities in skill levels (reading and math), seismic economic changes (widening wage gaps), sweeping demographic shifts (less education, lower skills). View summary
Adult Learners in a Changing America
I. Kirsch, M. Lennon, & C. Tamassia
in B. Guzzetti (Editor), Literacy for the New Millennium, Vol. 4, pp. 91–122 Publisher: Praeger
This ETS-authored article appears in a multivolume series on literacy research. View citation record
Learning a Living: First Results of the Adult Literacy and Life Skills Survey
Organisation for Economic Co-operation and Development & Statistics Canada
This report on the Adult Literacy and Life Skills survey (ALL) presents evidence on the nature and magnitude of the literacy gaps faced by OECD countries and how these gaps evolved over the previous decade. It contains interpretation of data for the seven countries participating in the first round of data collection. View the report here
A Human Capital Concern: The Literacy Proficiency of United States Immigrants
A. Sum, I. Kirsch, & K. Yamamoto
ETS Policy Information Report
In this report, the authors characterize the literacy proficiencies of U.S. residents born abroad and compares their performance not only with adults born in the U.S., but also with their foreign-born counterparts in other high-income countries around the world. View citation record
Pathways to Labor Market Success: The Literacy Proficiency of United States Adults
A. Sum I. Kirsch, & K. Yamamoto
ETS Policy Information Report
The report draws upon background and assessment data and information collected from the National Adult Literacy Survey (NALS) and the International Adult Literacy Survey (IALS). The authors find connections between the literacy skills of adults and their success in the labor market during the 1990s. View citation record
Digital Transformation: A Framework for ICT Literacy
International ICT Literacy Panel (2002)
This report describes the work of a panel that ETS convened in January 2001 to study the growing importance of existing and emerging information and communication technologies (ICT) and their relationship to literacy. The panel was made up of international experts from education, government, nongovernmental organizations (NGOs), labor and the private sector. View citation record
Reading for Change: Performance and Engagement Across Countries
I. Kirsch et al. (2002)
Publisher: Organisation for Economic Co-operation and Development
This report looks closely at the reading results of the first Program in International Student Assessment (PISA) survey. It shows reading literacy levels of students age 15, in 32 countries, in terms of how well they can use written materials to meet the challenges of the real world and to become lifelong learners. View citation record
Literacy in the Information Age: Final Report of the International Adult Literacy Survey
Organisation for Economic Co-operation and Development and Statistics Canada (2000)
This report looks at the literacy gaps faced by 20 OECD countries for which comparable household assessment results are included. It offers insights into the factors that influence the development of adult skills in various settings — at home, at work and across countries. View the report
Adult Literacy in OECD Countries: Technical Report on the First International Adult Literacy Survey
T. S. Murray, I. S. Kirsch, and L. Jenkins (Eds.) (1997)
National Center for Education Statistics
This publication provides detail on IALS, which extended educational measurement and assessment to the adult population. It describes detail on the design, development and statistical analysis of this assessment, conducted in eight countries. View publisher's page
Find a Publication
An ETS report.