Appendix B – PIAAC proficiency levels

Description of PIAAC proficiency levels on the literacy scale: 2012

Proficiency levels and cut scores for literacy Literacy task descriptions
Level 5 (376 – 500) At this level, tasks may require the respondent to search for and integrate information across multiple, dense texts; construct syntheses of similar and contrasting ideas or points of view; or evaluate evidence-based arguments. Application and evaluation of logical and conceptual models of ideas may be required to accomplish tasks. Evaluating reliability of evidentiary sources and selecting key information is frequently a requirement. Tasks often require respondents to be aware of subtle, rhetorical cues and to make high-level inferences or use specialized background knowledge.
Level 4 (326 – 375) Tasks at this level often require respondents to perform multiple-step operations to integrate, interpret, or synthesize information from complex or lengthy continuous, non-continuous, mixed, or multiple type texts. Complex inferences and application of background knowledge may be needed to perform the task successfully. Many tasks require identifying and understanding one or more specific, non-central idea(s) in the text in order to interpret or evaluate subtle evidence-claim or persuasive discourse relationships. Conditional information is frequently present in tasks at this level and must be taken into consideration by the respondent. Competing information is present and sometimes seemingly as prominent as correct information.
Level 3 (276 – 325) Texts at this level are often dense or lengthy, and include continuous, non-continuous, mixed, or multiple pages of text. Understanding text and rhetorical structures become more central to successfully completing tasks, especially navigating complex digital texts. Tasks require the respondent to identify, interpret, or evaluate one or more pieces of information, and often require varying levels of inference. Many tasks require the respondent to construct meaning across larger chunks of text or perform multi-step operations in order to identify and formulate responses. Often tasks also demand that the respondent disregard irrelevant or inappropriate content to answer accurately. Competing information is often present, but it is not more prominent than the correct information.
Level 2 (226 – 275) At this level, the medium of texts may be digital or printed, and texts may comprise continuous, non-continuous, or mixed types. Tasks at this level require respondents to make matches between the text and information, and may require paraphrasing or low-level inferences. Some competing pieces of information may be present. Some tasks require the respondent to
  • cycle through or integrate two or more pieces of information based on criteria;
  • compare and contrast or reason about information requested in the question; or
  • navigate within digital texts to access and identify information from various parts of a document.
Level 1 (176 – 225) Most of the tasks at this level require the respondent to read relatively short digital or print continuous, non-continuous, or mixed texts to locate a single piece of information that is identical to or synonymous with the information given in the question or directive. Some tasks, such as those involving non-continuous texts, may require the respondent to enter personal information onto a document. Little, if any, competing information is present. Some tasks may require simple cycling through more than one piece of information. Knowledge and skill in recognizing basic vocabulary, determining the meaning of sentences, and reading paragraphs of text is expected.
Below level 1 (0 – 175) The tasks at this level require the respondent to read brief texts on familiar topics to locate a single piece of specific information. There is seldom any competing information in the text and the requested information is identical in form to information in the question or directive. The respondent may be required to locate information in short continuous texts. However, in this case, the information can be located as if the text were non-continuous in format. Only basic vocabulary knowledge is required, and the reader is not required to understand the structure of sentences or paragraphs or make use of other text features. Tasks below level 1 do not make use of any features specific to digital texts.

Examples of literacy items

Items that exemplify the pertinent features of the proficiency levels in the domain of literacy are described below. In order to be consistent with the OECD international report, levels 4 and 5 are combined in the figures in this report (level 4/5).

Level 4: Library search (Item ID: C323P002)

Difficulty score: 348

The stimulus displays results from a bibliographic search from a simulated library website. The test-taker is asked to identify a book suggesting that the claims made both for and against genetically modified foods are unreliable. He or she needs to read the title and the description of each book in each of the entries reporting the results of the bibliographic search in order to identify the correct book. Many pieces of distracting information are present. The information that the relevant book suggests that the claims for and against genetically modified foods are unreliable must be inferred from the statement that the author “describes how both sides in this hotly contested debate have manufactured propaganda, tried to dupe the public and...[text ends].”

Level 3: Library search (Item ID: C323P003)

Difficulty score: 289

This task uses the same stimulus as the previous example. The test-taker is asked to identify the name of the author of a book called Ecomyth. To complete the task, the test-taker has to scroll through a list of bibliographic entries and find the name of the author specified under the book title. In addition to scrolling, the test-taker must be able to access the second page where Ecomyth is located by either clicking the page number (2) or the word “next”. There is considerable irrelevant information in each entry to this particular task, which adds to the complexity of the task.

Level 2: Lakeside fun run (Item ID: C322P002)

Difficulty score: 240

The stimulus is a simulated website containing information about the annual fun run/walk organized by the Lakeside community club. The test-taker is first directed to a page with several links, including “Contact Us” and “FAQs”. He or she is then asked to identify the link providing the phone number of organizers of the event. In order to answer this item correctly, the test-taker needs to click on the link “Contact Us”. This requires navigating through a digital text and some understanding of web conventions. While this task might be fairly simple for test-takers familiar with web-based texts, some respondents less familiar with web-based texts would need to make some inferences to identify the correct link.

Level 1: Generic medicine (Item ID: C309A321)

Difficulty score: 219

The stimulus is a short newspaper article entitled “Generic medicines: Not for the Swiss”. It has two paragraphs and a table in the middle displaying the market share of generic medicines in 14 European countries and the United States. The test-taker is asked to determine the number of countries in which the generic drug market accounts for 10% or more of total drug sales. The test-taker has to count the number of countries with a market share greater than 10%. The percentages are sorted in descending order to facilitate the search. The phrase “drug sales”, however, does not appear in the text; therefore, the test-taker needs to understand that “market share” is a synonym of “drug sales” in order to answer the question.

Below level 1: Election results (Item ID: C302BC02)

Difficulty score: 162

The stimulus consists of a short report of the results of a union election containing several brief paragraphs and a simple table identifying the three candidates in the election and the number of votes they received. The test-taker is asked to identify which candidate received the fewest votes. He or she needs to compare the number of votes that the three candidates received and identify the name of the candidate who received the fewest votes. The word “votes” appears in both the question and in the table and nowhere else in the text.

Description of PIAAC proficiency levels on the numeracy scale: 2012

Proficiency levels and cut scores for numeracy Numeracy task descriptions
Level 5 (376 – 500) Tasks at this level require the respondent to understand complex representations and abstract and formal mathematical and statistical ideas, possibly embedded in complex texts. Respondents may have to integrate multiple types of mathematical information where considerable translation or interpretation is required; draw inferences; develop or work with mathematical arguments or models; and justify, evaluate and critically reflect upon solutions or choices.
Level 4 (326 – 375) Tasks at this level require the respondent to understand a broad range of mathematical information that may be complex, abstract or embedded in unfamiliar contexts. These tasks involve undertaking multiple steps and choosing relevant problem- solving strategies and processes. Tasks tend to require analysis and more complex reasoning about quantities and data; statistics and chance; spatial relationships; and change, proportions and formulas. Tasks at this level may also require understanding arguments or communicating well-reasoned explanations for answers or choices.
Level 3 (276 – 325) Tasks at this level require the respondent to understand mathematical information that may be less explicit, embedded in contexts that are not always familiar and represented in more complex ways. Tasks require several steps and may involve the choice of problem-solving strategies and relevant processes. Tasks tend to require the application of number sense and spatial sense; recognizing and working with mathematical relationships, patterns, and proportions expressed in verbal or numerical form; and interpretation and basic analysis of data and statistics in texts, tables and graphs.
Level 2 (226 – 275) Tasks at this level require the respondent to identify and act on mathematical information and ideas embedded in a range of common contexts where the mathematical content is fairly explicit or visual with relatively few distractors. Tasks tend to require the application of two or more steps or processes involving calculation with whole numbers and common decimals, percents and fractions; simple measurement and spatial representation; estimation; and interpretation of relatively simple data and statistics in texts, tables and graphs.
Level 1 (176 – 225) Tasks at this level require the respondent to carry out basic mathematical processes in common, concrete contexts where the mathematical content is explicit with little text and minimal distractors. Tasks usually require one-step or simple processes involving counting, sorting, performing basic arithmetic operations, understanding simple percents such as 50%, and locating and identifying elements of simple or common graphical or spatial representations.
Below level 1 (0 – 175) Tasks at this level require the respondents to carry out simple processes such as counting, sorting, performing basic arithmetic operations with whole numbers or money, or recognizing common spatial representations in concrete, familiar contexts where the mathematical content is explicit with little or no text or distractors.

Examples of numeracy items

Items that exemplify the pertinent features of the proficiency levels in the domain of numeracy are described below. In order to be consistent with the OECD international report, levels 4 and 5 are combined in the figures in this report (level 4/5). No items mapped at level 5 in numeracy.

Level 4: Education level (Item ID: C632P001)

Difficulty score: 354

The stimulus for this item consists of two stacked-column bar graphs presenting the distribution of the Mexican population by years of schooling for men and women separately. The y-axis of each of the graphs is labeled “percentage” with 6 grid lines labeled “0%”, “20%”, “40%”, “60%”, “80%”, and “100%”. The x-axis is labeled “year” and data are presented for 1960, 1970, 1990, 2000, and 2005. A legend identifies three categories of schooling: “more than 6 years of schooling”, “up to 6 years of schooling”, and “no schooling”. The test-taker is asked to approximate what percentage of men in Mexico had more than 6 years of schooling in 1970, choosing from a pull-down menu that has 10 response categories: “0-10%”, “10-20%”, and so on.

Level 3: Package (Item ID: C657P001)

Difficulty score: 315

The stimulus for this item consists of an illustration of a box constructed from folded cardboard. The dimensions of the cardboard base are identified. The test-taker is asked to identify which plan best represents the assembled box out of four plans presented in the stimulus.

Level 2: Logbook (Item ID: C613A520)

Difficulty score: 250

The stimulus for this item consists of a page from a motor vehicle logbook with columns for the date of the trip (start and finish), the purpose of the trip, the odometer reading (start and finish), the distance travelled, the date of entry and the driver’s name and signature. For the first date of travel (June 5), the column for the distance travelled is completed. The instructions inform the test-taker that “a salesman drives his own car and must keep a record of the miles he travels in a Motor Vehicle Log. When he travels, his employer pays him $0.35 per mile plus $40.00 per day for various costs such as meals.” The test-taker is asked to calculate how much he will be paid for the trip on June 5.

Level 1: Candles (Item ID: C615A602)

Difficulty score: 221

The stimulus for this item consists of a photo of a box containing tea light candles. The packaging identifies the product (tea light candles), the number of candles in the box (105 candles) and its weight. While the packaging partially covers the top layer of candles, it can be seen that the candles are packed in five rows of seven candles each. The instructions inform the test-taker that there are 105 candles in a box and asks him or her to calculate how many layers of tea candles are packed in the box.

Below level 1: Price tag (Item ID: C602A501)

Difficulty score: 168

The stimulus for this item consists of four supermarket price tags. These identify the product, the price per pound, the net weight, the date packed and the total price. The test-taker is asked to indicate the item that was packed first by simply comparing the dates on the price tags.

Description of PIAAC proficiency levels on the problem solving in technology-rich environments scale: 2012

Proficiency levels and cut scores for problem solving in technology-rich environments Problem solving in technology-rich environments task descriptions
Level 3 (341 – 500) At this level, tasks typically require the use of both generic and more specific technology applications. Some navigation across pages and applications is required to solve the problem. The use of tools (e.g., a sort function) is required to make progress towards the solution. The task may involve multiple steps and operators. The goal of the problem may have to be defined by the respondent, and the criteria to be met may or may not be explicit. There are typically high monitoring demands. Unexpected outcomes and impasses are likely to occur. The task may require evaluating the relevance and reliability of information in order to discard distractors. Integration and inferential reasoning may be needed to a large extent.
Level 2 (291 – 340) At this level, tasks typically require the use of both generic and more specific technology applications. For instance, the respondent may have to make use of a novel online form. Some navigation across pages and applications is required to solve the problem. The use of tools (e.g., a sort function) can facilitate the resolution of the problem. The task may involve multiple steps and operators. The goal of the problem may have to be defined by the respondent, though the criteria to be met are explicit. There are higher monitoring demands. Some unexpected outcomes or impasses may appear. The task may require evaluating the relevance of a set of items to discard distractors. Some integration and inferential reasoning may be needed.
Level 1 (241 – 290) At this level, tasks typically require the use of widely available and familiar technology applications, such as e-mail software or a web browser. There is little or no navigation required to access the information or commands required to solve the problem. The problem may be solved regardless of the respondent’s awareness and use of specific tools and functions (e.g., a sort function). The tasks involve few steps and a minimal number of operators. At the cognitive level, the respondent can readily infer the goal from the task statement; problem resolution requires the respondent to apply explicit criteria; and there are few monitoring demands (e.g., the respondent does not have to check whether he or she has used the appropriate procedure or made progress towards the solution). Identifying content and operators can be done through simple match. Only simple forms of reasoning, such as assigning items to categories, are required; there is no need to contrast or integrate information.
Below level 1 (0 – 240) Tasks are based on well-defined problems involving the use of only one function within a generic interface to meet one explicit criterion without any categorical or inferential reasoning, or transforming of information. Few steps are required and no sub-goal has to be generated.

Examples of problem solving in technology-rich environments items

Items that exemplify the pertinent features of the proficiency levels in the domain of problem solving in technology-rich environments are described below.

Level 3: Meeting rooms (Item ID: U02)

Difficulty score: 346

This task involves managing requests to reserve a meeting room on a particular date using a reservation system. Upon discovering that one of the reservation requests cannot be accommodated, the test-taker has to send an e-mail message declining the request. Successfully completing the task involves taking into account multiple constraints (e.g., the number of rooms available and existing reservations). Impasses exist, as the initial constraints generate a conflict (one of the demands for a room reservation cannot be satisfied). The impasse has to be resolved by initiating a new sub-goal, i.e., issuing a standard message to decline one of the requests. Two applications are present in the environment: an e-mail interface with a number of e-mails stored in an inbox containing the room reservation requests, and a web-based reservation tool that allows the user to assign rooms to meetings at certain times. The item requires the test-taker to “Use information from a novel web application and several e-mail messages, establish and apply criteria to solve a scheduling problem where an impasse must be resolved, and communicate the outcome.” The task involves multiple applications, a large number of steps, a built-in impasse, and the discovery and use of ad hoc commands in a novel environment. The test-taker has to establish a plan and monitor its implementation in order to minimize the number of conflicts. In addition, the test-taker has to transfer information from one application (e-mail) to another (the room-reservation tool).

Level 2: Club membership (Item ID: U19b)

Difficulty score: 296

This task involves responding to a request for information by locating information in a spreadsheet and e-mailing the requested information to the person who asked for it. The test-taker is presented with a word-processor page containing a request to identify members of a bike club who meet two conditions, and a spreadsheet containing 200 entries in which the relevant information can be found. The required information has to be extracted by using a sort function. The item requires the test-taker to “Organize large amounts of information in a multiple-column spreadsheet using multiple explicit criteria and locate and mark relevant entries.” The task requires switching between two different applications and involves multiple steps and operators. It also requires some amount of monitoring. Making use of the available tools greatly facilitates identifying the relevant entries.

Level 1: Party invitations (Item ID: U01A)

Difficulty score: 286

This task involves sorting e-mails into pre-existing folders. An e-mail interface is presented with five e-mails in an Inbox. These e-mails are responses to a party invitation. The test-taker is asked to place the response e-mails into a pre-existing folder to keep track of who can and cannot attend a party. The item requires the test-taker to “Categorize a small number of messages in an e-mail application in existing folders according to a single criterion.” The task is performed in a single and familiar environment and the goal is explicitly stated in operational terms. Solving the problem requires a relatively small number of steps and the use of a restricted range of operators and does not demand a significant amount of monitoring across a large number of actions.

APPENDIX A – BACKGROUND ON PIAAC
APPENDIX C – DATA TABLES