Innovations in PIAAC: How Technology and Methodology Differentiate PIAAC

Matthias von Davier, Educational Testing Service
Jean Francois Rouet, University of Poitiers

The Programme for the International Assessment of Adult Competencies (PIAAC) is the first major international assessment predominantly administered using computer-based testing. Among other benefits, the computer delivery mode allowed the recording of online process data in addition to correct or incorrect responses. These data include the time taken to complete each item, the steps taken (i.e., selection of commands or menu items, access to information pages and so forth), and the use of tools in the test of problem solving in technology-rich environments.

Online process data have the potential to supplement traditional indicators in order to provide more precise and accurate estimates of proficiency. However, general models linking online indicators to performance are still lacking. The large set of data collected as part of PIAAC provides an ideal basis for the exploration and benchmarking of such indicators.

In the first part of the talk, we will discuss innovations in PIAAC as the first computer-based large scale international assessment of skills. We will examine the use of process and timing data as well as free response data collected using innovative item formats and will show how modern psychometric methodologies can be used to gain a deeper understanding of the proficiency and background data collected using information technology.

The second part of the talk will give an overview of different avenues of exploratory as well as theory-driven research using these new sources of information on test performance. Advances in theoretical conceptualizations of problem solving will be exemplified and connected with innovative methodologies used to analyze this complex database.

Estimated Returns to Quality-Adjusted Education: New Evidence from PIAAC

Guido Schwerdt, Ifo Institute for Economic Research, Germany

An overwhelming number of studies provide evidence that human capital is a key driver of individual and national economic growth, but it is less than clear how to measure it. Quantity-based measures, such as years of schooling, dominate the empirical literature on the returns to human capital accumulation, but the quality of schooling might change over time and vary across education systems, which reduce the comparability of the estimated returns to schooling across cohorts and countries. Based on data from PIAAC, we provide new evidence on the returns to quality-adjusted education following the methodology pioneered in Hanushek and Zhang (2009). This paper adds to the literature in two important ways: First, it provides internationally comparable estimates of returns to quality-adjusted education for a larger set of countries than previously available. Second, the quality-adjustment is based on new domains of skills that are likely to be more accurate and reliable measures of cognitive skills relevant in the labor markets of modern information societies.

Understanding How Skills are Used at Work

Glenda Quintini, Organisation for Economic Co-operation and Development (OECD)

This paper explores how cognitive and generic skills are used at work and whether skills use — as opposed to skills proficiency alone — matters for labor productivity and wages. The paper also investigates the incidence and consequences of vertical and horizontal mismatch in the labor market, including their effect on wages and skills use at work.

The Acquisition and Decline of Literacy over the Course of Life

Rolf K. W. van der Velden, Maastricht University

I will look at different components of literacy and how they develop over the course of life. I will show that these components are differentially affected by age. The components of literacy are also differentially related to factors that are assumed to affect skills acquisition and skills decline. Moreover, they have divergent effects on economic and noneconomic outcomes. I will show under what conditions mismatches in literacy components affect labor market outcomes.

Better Skills, Better Jobs, Better Lives

Andreas Schleicher, Organisation for Economic Co-operation and Development (OECD)

We’ve all been in the economic crisis long enough to see there is no simple way to bail ourselves out of it through economic stimulus or by simply printing more money. Countries’ best bet is to give more people better skills to compete, collaborate, and connect in ways that drive our economies forward. Over the last decade, well over half of the productivity increase in the industrialized world has been driven by improved education. So jobs and growth these days are mainly about working smarter. The alternative is clear: Without the right skills, people are left on the margins of society, technological progress does not translate into economic growth, and countries can’t compete globally.

Using PIAAC Data for Producing Regional Estimates

Kentaro Yamamoto, Educational Testing Service (ETS)

While the assessment design for PIAAC yielded a high-quality dataset, there is strong interest in the development and use of other methodologies when it is too costly or not feasible to administer direct measures to all populations of interest. This paper explores the feasibility of using model-based small area estimation procedures to compare the accuracy of estimating proficiency distributions under various conditions where only partial cognitive and background information are available. PIAAC combines the use of item response theory (IRT) with population modeling to efficiently obtain reliable and valid estimates of a distribution of proficiencies in policy-relevant subgroups. The methodology capitalizes on the relationship between a set of background variables, cognitive proficiencies and Bayesian statistical methodology that combines the prior distribution and likelihood function of individual respondents.

The similarities of this methodology to model-based small area estimates are striking and will be evaluated with respect to providing a set of tools under different data configurations — from the “full survey setting,” which provides for the most accurate estimate, to the “background questionnaire only setting,” which allows model-based estimates based on the multivariate distribution of the target population, to the “census only data setting,” which allows for a much less precise estimate because even the multivariate dependencies between census variables may be borrowed from “donor” populations that collected such data. The utility of this methodology in countries that have participated in PIAAC will be discussed.

The Relationship of Problem Solving to Literacy and Numeracy: Insights from PIAAC

Henry Braun, Boston College

The recently completed, multinational Programme for the International Assessment of Adult Competencies (PIAAC) incorporated a number of innovations, including computer administration and the assessment of problem solving in a technology-rich environment. Accordingly, there is particular interest in examining the relationship of problem solving to literacy and numeracy, as well as how that relationship is mediated by different background factors.

We propose to carry out an exploratory study of these relationships using quantile regression (QR) methods. In contrast to the standard least squares approach (which focuses on conditional expectations and assumes homoscedasticity), this approach makes weaker assumptions and will enable us to describe in detail how the distribution of problem solving scores varies as a function of both other competencies and background variables. We will model different quantiles of the conditional problem solving distribution using both parametric (linear regression) and nonparametric (spline function) strategies. Implications of the findings for policy will be discussed. The study will be carried out for four countries representing different types of education systems.