Human Capital and International Large-Scale Assessment

As societies become increasingly advanced, adults are expected to use information in more complex ways and to maintain and enhance their skills in literacy, numeracy, science and problem-solving in technology-rich environments throughout their lifetimes. Such skills are critical not only for personal achievement, but also for positive social, educational and economic outcomes around the world.

ETS’s research in this area is related to international large-scale comparative surveys of adults and youth of school age. The focus is on core skills such as reading, math/numeracy and science literacy. Other areas addressed in these surveys include innovative constructs such as financial literacy, health literacy and problem solving in technology-rich environments.

Projects

ETS’s research in human capital and international large-scale assessment takes place both in the United States and around the world. Our large-scale international comparative surveys are designed to meet policy needs reflecting a growing interest in skills as human capital. These are some examples:

OECD Survey of Adult Skills
ETS manages this major assessment for the Organisation for Economic Co-operation and Development (OECD) as part of the Programme for the International Assessment of Adult Competencies (PIAAC). This is the first large-scale, computer-delivered comparative assessment of adults in the world. It provides policymakers with a profile of adults’ knowledge, skills and competencies. PIAAC expands upon earlier literacy assessments, and it is the first multistage adaptive test used in large-scale comparative assessment.

We also work on two other projects related to the PIAAC program:

  • Education and Skills Online — A new computer-delivered assessment linked to PIAAC. It provides individual results for persons who want to demonstrate readiness skills, and allows institutions, organizations and local governments to assess a population for training and research purposes. Scheduled to be available in early 2015.
  • The World Bank’s Skills Toward Employment and Productivity (STEP) — This program, which is part of PIAAC, measures reading literacy and component skills. It seeks to help countries assess how different skill sets affect labor-market opportunities, and provides information on employability and marketability.

In November 2013, ETS organized the PIAAC Invitational Research Conference, which highlighted the importance of the PIAAC results for understanding the role of skills in a global context. Watch the conference’s opening video.

Programme for International Student Assessment (PISA)
ETS leads this global evaluation of education systems. PISA is a multinational survey of 15-year-old students, currently administered in some 70 countries. The test, which has transitioned from paper to computers, assesses what students know and their ability to apply their knowledge in real-life scenarios at an age where they begin to fully participate in society.

TIMSS and PIRLS
ETS acts as consultant to Boston College on two international assessments: Trends in International Mathematics and Science Study, as well as the Progress in International Reading Literacy Study.

IEA-ETS Research Institute (IERI)
ETS collaborates with colleagues from around the world on international large-scale assessments. One such example is the IEA-ETS Research Institute (IERI) — a partnership between ETS and the International Association for the Evaluation of Educational Achievement (IEA). IERI publishes an international journal and runs workshops on methodology using existing large-scale assessment databases. Both independently and as part of its involvement in IERI, ETS’s researchers also produce research reports, books and book chapters related to human capital and international large-scale assessment (see list below).

Featured Publications

Here is a selection of significant publications related to human capital and international large-scale assessment:

2014
2013
2012
2011

  • An Experimental Study of the Effects of Monetary Incentives on Performance on the 12th-Grade NAEP Reading Assessment
    H. Braun, I. Kirsch, & K. Yamamoto
    Teachers College Record, Vol. 113, No. 11, pp. 2309–2344

    This article reports on a study of monetary incentives for 12th-grade students taking an assessment designed to replicate the NAEP reading test. The study concluded that NAEP may both underestimate the reading abilities of students enrolled in 12th grade and yield biased estimates of certain achievement gaps. View publisher's page

  • Measuring Growth in a Longitudinal Large Scale Assessment with a General Latent Variable Model
    M. von Davier, X. Xu, & C. H. Carstensen
    Psychometrika, Vol. 76, No. 2, pp. 318–336

    This article presents approaches for use with longitudinal large-scale assessment data, i.e., data that were repeatedly collected on the same sample of students or adults. The article describes models that made it possible to examine whether the Matthew-effect (those who have more, gain more) can be identified in a national extension of the PISA assessment. View citation record

  • Investigation of Model Fit and Score Scale Comparability in International Assessments
    M. E. Oliveri & M. von Davier
    Psychological Test and Assessment Modeling, Vol. 53, No. 3, pp. 315–333

    This article presents evidence on scale stability and the need to account for national variations in how some tasks in international assessments are understood. While rigorous translation, verification and assessments of cultural appropriateness reduce the threat of cross-country differences in how tasks are understood, it is always possible to detect deviations from the international standard on a limited number of tasks. This article presents an approach of how such deviation can be remedied. View article

2010

  • Relationships Among Reading Skills of Adults with Low Literacy
    J. P. Sabatini, Y. Sawaki, J. R. Shore, & H. S. Scarborough
    Journal of Learning Disabilities, Vol. 43, No. 2, pp. 122–138

    In this study, researchers used a method known as confirmatory factor analysis to examine the relationship between literacy and several component reading skills (e.g., word recognition, fluency, language comprehension and vocabulary skills) in adults. View publisher's page

2009
2008

  • Preparing for an Epidemic of Limited Health Literacy: Weathering the Perfect Storm
    R. M. Parker, M. S. Wolf, & I. Kirsch
    Journal of General Internal Medicine, Vol. 23, No. 8, pp. 1273–1276

    Research has demonstrated strong links between literacy and health. ETS's recent report America's Perfect Storm considers the great risks the United States faces as a result of declining adult literacy, shifting demographics and a changing economy. It is essential to understand how these changes will impact health care. View publisher's page

2007

  • America's Perfect Storm: Three Forces Changing Our Nation's Future
    I. Kirsch, H. Braun, K. Yamamoto, & A. Sum
    ETS Policy Information Report

    This report looks at the convergence of three powerful sociological and economical forces that are changing the future of the United States: substantial disparities in skill levels (reading and math), seismic economic changes (widening wage gaps), sweeping demographic shifts (less education, lower skills). View summary

  • Adult Learners in a Changing America
    I. Kirsch, M. Lennon, & C. Tamassia
    in B. Guzzetti (Editor), Literacy for the New Millennium, Vol. 4, pp. 91–122 Publisher: Praeger

    This ETS-authored article appears in a multivolume series on literacy research. View citation record

2005

  • Learning a Living: First Results of the Adult Literacy and Life Skills Survey
    Organisation for Economic Co-operation and Development & Statistics Canada

    This report on the Adult Literacy and Life Skills survey (ALL) presents evidence on the nature and magnitude of the literacy gaps faced by OECD countries and how these gaps evolved over the previous decade. It contains interpretation of data for the seven countries participating in the first round of data collection. View the report here

2004
1997–2002