Key Insights and Their Implications for States and Districts
Insights gleaned from the body of work developed under RfU have important implications for stakeholders in states and districts across the nation. These lessons learned can meaningfully inform literacy efforts designed to ensure that those lacking opportunity can gain the skills needed.
The insights and recommendations that follow are those of the authors, gleaned from the empirical evidence and results accumulated across years of work and scholarship on the RfU initiative, and focus on those outcomes that are most relevant to state and district leaders. We divide the report into sections based on the major strands that were used to organize the National Symposium on Reading for Understanding:
We start each section with a brief summary of the research undertaken by the RfU teams. We then provide recommendations for state and district leaders based on the key insights gained from the six research projects.
Monitoring Progress: A long-term action agenda employing a continuous improvement strategy requires the establishment of a set of milestones representing intermediate goals and the construction of a set of indicators that can be used to monitor progress toward those goals. As Richard Reeves explains, 'Indicators are necessary to guide policy, drive data collection strategies, and measure progress.' Ideally, the indicators will incorporate both quantitative and qualitative evidence.
Kirsch et al., Choosing Our Future
We begin with assessment because it represents the starting and culminating point of any learning and instructional cycle and, if properly conceived, can also be used formatively to guide and enhance the instructional program. In other words, we see assessment serving not only as a progress monitoring function but also as a catalyst and contributor to policy and structural change.
The work of the ETS-led assessment team focused on two types of assessment: scenario-based assessments (SBA), which require the application of complex reading comprehension skills aligned with our framework for a twenty-first-century reading literacy construct,36 and component assessments, which measure discrete foundational reading skills (see www.ets.org/research/topics/reading_for_understanding/ for more information).37
Each of the other five research teams also developed assessments, primarily classroom-based ones such as quizzes and discussion questions, which targeted skills in which they saw gaps (e.g., academic language, perspective taking, inference making, progress monitoring, and so forth). Contrary to the common complaint that assessment is an add-on that takes time away from instruction, these projects found that a combination of carefully designed and utilized assessments can facilitate, guide, and enable effective instructional approaches and accelerate learning.38
Multiple empirical studies have found evidence to suggest that some percentage of students continue to struggle mastering foundational skills through to secondary school, thwarting learning from text in the content areas.39 Here we refer to foundational skills as the set of subskills that are necessary to "get the words off the page," including decoding, word recognition, and reading fluency. Component assessments that measure foundational skills complement higher level comprehension measures by helping teachers to distinguish weakness in component skills from weaknesses in higher level comprehension processes. This distinction can lead to a better alignment between students' needs and instruction. Providing both types of assessments gives teachers knowledge of the origin of the difficulties students have so teaching strategies can be directed toward remediating the problem. It should be noted that while both types of assessments will benefit English language learners, new assessments specifically targeted to second-language needs should be developed to focus instruction. Below, we focus on specific recommendations based on the RfU assessment research and development. In particular we focus on issues related to the match between assessments and instruction; the importance of goal-directed activities, assessment frameworks and transparency; the importance of foundational skills; the role of motivation and background knowledge on score interpretation; the inclusion of disciplinary and multiple source reading; and prioritizing web-based assessments.
Recommendations for state and district leaders regarding assessment:
- Provide assessment activities that look like quality learning and instruction—in other words, tests worth teaching to. Good teaching practices should align with the processes required in assessment tasks.40 While preparing students for taking state reading literacy tests is a necessary requirement, the measurement priority during a high-stakes assessment currently is to sample skills broadly, not for students to learn the text content in order to meet a learning goal or reason about an issue. However, in school or classroom assessments, it is a priority that students learn from and reason about what they read, as well as practice the skills and strategies necessary to achieve complex goals. In statewide summative assessments, as well as those used at the district, school, and classroom levels, tasks designed to mimic quality learning activities will provide meaningful guidance for tailoring curriculum and instruction across grades and school subjects.41 Yet one of the barriers to this potential reform is the possible firewall between summative assessment and instruction. Summative tests are traditionally designed to measure student performance, not support it. It is assumed that any support, learning, or task authenticity might contaminate a student's score. However, if the goal of instruction is to help students grow and meet life's challenges, then why should the tasks used to measure reading be a distal proxy of how a person would read in real life? In a similar vein, skilled readers use reading strategies, so why not include strategies on a reading test to encourage their use and serve as a model for instruction? If properly designed, assessments can model good reading practices while providing solid measurement of a student's reading ability.42
- Provide assessment activities that include complex, goal-directed tasks, in which academic learning from text is a primary goal. Answering isolated questions about a single passage is not the same as learning and reasoning about content knowledge from text or other information sources such as images, charts, maps, and videos.43 The goal of most reading in school is to learn new subject knowledge. A second common goal, in and out of school, is to learn enough about a topic or issue to make a decision, solve a problem, understand a point of view, or argue a point (e.g., should I apply to this academic program or this job?).44 For learning from text to occur, text content must be integrated with one's prior knowledge, creating new knowledge.45 Further, providing a complex goal (e.g., see Community Garden example) demands of students that they allocate attention to appropriate skills and strategies necessary to the task46 (see Curriculum and Instruction section below for further discussion). Without an overarching goal for reading, students in a testing context may fail to read the entire text and instead adopt test-taking strategies that do not reflect construct-relevant processing.47 As such, we argue that reading assessments should provide meaningful goals for reading that help the reader engage in deep processing.48
- Demand transparency in what the assessment measures and the level of performance expected—not merely reliable and valid score reports. This information is essential to ensuring that teaching and testing are aligned. Test scores are part of an assessment's value, but educators also need to understand the construct (what the test was designed to measure) and the test design (how the items and tasks are designed to measure the construct) in order to make effective use of the results to tailor instruction. Moreover, developers of curricula and professional development materials can and should use this information to create well-aligned materials. Thus, we recommend that all assessments of reading should have a framework that describes the construct and how it is measured. This is akin to evidence-centered design,49 whereby the test design and construction process is documented for both transparency reasons and for the empirical support for key constructs and design decisions. This way, educators can make more informed decisions as to whether a given assessment meets their needs and how to plan instructional decisions accordingly.
- Make the identification of students with foundational skill weaknesses in reading a priority, especially beyond grade 4. Component assessments are designed for this purpose. Identifying the proportion of at-risk or struggling readers is a key step needed for district and school level planning for professional development, interventions, and associated policy decisions. Reading assessments should measure foundational reading skills in tandem with higher-level reading comprehension skills through early secondary schooling, or until foundational skills are commensurate with grade level expectations, in order to identify at risk or struggling students. Contrary to some expectations, foundational skills may not be adequately developed after fourth grade, thus continuing to negatively impact higher level comprehension processes.50 We therefore recommend that educators continue to monitor the adequacy of foundational skills when problems exist with students' comprehension.
- Make users of test scores aware of how motivation and background knowledge mediate assessment performance and score interpretation. Users cannot automatically assume that every student score is a true estimate of that student's reading ability. This is because there are a range of other factors that might explain a student's test score that are not directly related to the construct. For instance, low or high knowledge of topics presented on a test can result in misinterpretations of the strengths or weaknesses in comprehension skills underlying that performance.51 Further, motivation plays a role in testing,52 and the instructional approach one recommends may differ significantly based on whether the test taker expends high or low effort when taking the test, or if other emotional states (e.g., anxiety) impact performance. When feasible, we recommend using methods for measuring knowledge or motivation during a test to estimate the impact of these critical comprehension correlates. These actions may improve score interpretation and subsequent decisions for alternative courses of instruction.
- Make the inclusion of disciplinary and multiple source reading in assessments a priority. There is a significant mismatch in the passages and questions used in current assessments versus the variety of comprehension skills required to learn from text in academic (and nonacademic) environments. Historically, passages on reading comprehension tests tended not to include content-area reading texts so as to reduce overlap with the subject areas and to reduce the impact of background knowledge on comprehension. However, the authors of the Common Core State Standards and others have argued for the importance of content area and disciplinary reading in twenty-first-century reading environments. In short, the cognitive strategies used to understand and learn from a science article differ from those applied to a historical document or a novel.53 For instance, historians may value primary over secondary evidence when developing a causal theory for a set of events, while scientists may weigh the amount and type of empirical evidence when choosing to adopt one theory over another. In short, what it means to read successfully in one domain may differ depending upon the social expectations of the members of the discipline. Assessments should capture this fundamental difference across academic domains. Further, given the proliferation of information on the internet, the ability to evaluate credibility, understand multiple perspectives, and corroborate information across sources is an essential twenty-first-century skill that should be assessed.54 It is no longer possible to accept what is written to be true, nor is it often possible to get the information one needs in a single source. Thus, students must now be selective in what they read, be critical about it, and integrate the multiple sources based on the supporting evidence. These skills should be taught in the classroom and measured in the next generation of assessments.
- Prioritize computer- or web-based assessments, because most reading and learning in the world and in the future will take place on electronic devices. Paper-only reading comprehension is inadequate preparation for the workplace and postsecondary technology-based learning environments in today's society. While people still read print-based materials, the construct has changed and involves the use of digital sources and the associated affordances they provide.55 For instance, the internet offers a wide range of sources that have to be searched and sorted for relevance. This requires a knowledge not only of how to effectively use search engines but how to scroll, use hyperlinks, and so forth. Digital literacy also involves an understanding of synchronous (e.g., online chat in real time) and asynchronous forms of communication (a message board updated two years ago). Both forms of digital communication involve keeping track of what was said, what questions were asked, which ones were answered, and those needing a response. Such multitasking could potentially be very demanding on time management and the ability to notice errors. Beyond the changes in construct, computer-assisted assessment affords multiple advantages in administration, construct coverage, cost and time efficiency, and scoring that are impossible to mimic in paper-only assessments. This will also necessitate that states, in partnership with districts, ensure that all students have access to technology within the learning environment in order to develop digital literacy skills. The increased access should not only help with lowering long-term assessment costs but to ensure that students are prepared to read in twenty-first-century technology-rich environments.
Curriculum and Instruction
All too often, interventions are applied at single transition points in an individual's life. Examples include preschool programs to foster cognitive development or career readiness curricula for high school seniors. These efforts may be quite effective according to certain indicators but, too often, their long-term impact is attenuated. The reason is simple: Human and social capital are not built at a single point in time but, rather, are accumulated over many years as the result of a range of experiences and interactions across multiple contexts.
Kirsch et al., Choosing Our Future
As noted, five RfU teams tackled the challenge of building curriculum and instruction materials, then implementing them, spanning preK-12. We note that no RfU team started with the premise of teaching to the Common Core State Standards or other new college- and career-readiness standards,56 though the instructional programs produced would, in retrospect, be found to be consistent with these standards.
Synthesizing insights for curriculum and instruction across all of the RfU projects in just a few pages clearly requires oversimplification of the richness of their accomplishments. However, we can codify a few key, priority insights. In doing so, we first need to clarify our terminology. In common usage, the terms reading and comprehension are sometimes used interchangeably and sometimes to distinguish sets of skills. Reading ability often is used to mean what we have called foundational skills, that is, the set of visual processing and basic language skills that get the words off the page into one's head. These skills include decoding and word recognition, as well as ordinary command of spoken language vocabulary and syntax, which together culminate in fluent reading. If there is an assumption of understanding or comprehension, it is usually at a basic or literal comprehension level.
Comprehension can be thought of as something simple, like literal recall or recognition of the gist of what one just read. But this is not what is meant or measured in reading comprehension tests across grades. We would not have nearly 30 percent of students below basic comprehension levels by this simple definition. Rather, comprehension refers to a more complex construct of understanding what one reads, and that is what is measured on assessments. At a minimum, this construct of comprehension requires sophisticated language processing (complex, written forms of vocabulary and grammar), inference, reasoning, perspective taking, and interpretation. It requires regulating a set of skills and strategies toward goals of building new knowledge, making a decision, solving a problem, or applying what one learns. Note that this description of comprehension applies to reading (visual processing of texts), listening, or comprehending any mixture of media (e.g., animation, film, etc.). While the RfU teams differed in the specific definitions of comprehension that they used to guide their research projects, they would all agree that some mixture of these higher order processes are required to read for understanding.
With this in mind, we highlight four common misconceptions that should be addressed by the education community.
- Reading (i.e., print-based foundational skills) should be the sole or even primary focus of early elementary reading instruction (preK-4). While foundational skills (e.g., decoding and reading fluency) must continue to be a priority, comprehension development is an equivalent priority and requires a focus on oral language comprehension and development. Individual differences in vocabulary and the sophistication of language skills vary widely among children already at preschool. This gap needs to be addressed as soon as formal schooling begins and continue to be addressed until the gap is narrowed.57
- (Reading) comprehension instruction is complete by grade 4 or 5. We have long sent a mixed message to students (and teachers) by treating reading comprehension as if it is a skill set that is mastered in elementary school, even as we set content standards and outcome tests of reading literacy through high school. The complex construct of reading comprehension or reading for understanding as measured from preK-12 needs to continue to be explicitly developed through middle and high school, addressing the varying cognitive strategies applied to reading in each discipline.58
- Literacy involves only the ability to gain knowledge or information through text. The research conducted and reviewed through the RfU projects demonstrates that the application of language, academic vocabulary, reasoning, and the social contexts of literacy are essential elements to developing and applying literacy skills.59
- Reading is an activity involving one reader and one text. While often thought of in that way, comprehension denotes more than this solitary interaction; it is a social activity that involves listening skills, the sharing and co-constructing of ideas, and perspective taking. Rich literacy environments involve students and teachers interacting to scaffold understanding and provide feedback on each other's ideas. An understanding and appreciation of varying points of view and perspectives are part of the interpretive fabric of reading for understanding.60
To address these misconceptions, we structure our recommendations for state and district leaders around three areas: instructional focus on reading comprehension, preK-12; the integration of complex literacy skills; and greater emphasis on reading to learn.
Recommendations for state and district leaders regarding curriculum and instruction:
- Ensure that early elementary and preschool curricula focus on language development, comprehension, and knowledge building, as well as foundational reading skills. Whereas foundational skills (e.g., decoding and reading fluency) must continue to be a priority, comprehension development is an equivalent priority. This comprehension development is mediated through oral language and visual media. Individual differences across children in vocabulary and the sophistication of language skills are wide, even at the start of schooling.61 Enhancing language comprehension should include building academic content knowledge in science, social studies, and so forth. If children are more sophisticated users and producers of language, with adequate knowledge of academic domains, this may mediate the so-called fourth-grade slump seen in reading comprehension tests.
- Ensure that schools have the resources and policies that enable middle and high schools to address the needs of students who fail to achieve mastery of foundational reading skills. Most students who fail to achieve comprehension also show only adequate-to-weak foundational skills.62 Many such students may not have sufficiently poor skills to be classified as reading or learning disabled, but they are likely reading well below average for their grade level. Students with these levels of skills often read slowly and are nonfluent when reading grade-level texts. They do not recognize or decode words automatically and are slow to learn new vocabulary from texts on their own. They expend their attention and memory resources on these lower-level processes at the expense of higher level comprehension and reasoning processes. They are reluctant readers; they do not read widely or frequently on their own. State and district assistance is needed when a high prevalence of such students are clustered in middle and high schools, because those schools may not have structures and expertise in place to intervene or remediate foundational skills and comprehension skills and to meet curriculum content standards simultaneously at scale.
- Ensure an instructional focus on reading comprehension beyond elementary and through secondary schooling. As students progress through middle and high school, instruction should include the specialized ways of reading, thinking, and conveying information needed for each of the content areas. This implies that content area teachers at all grade levels should include these reading comprehension strategies in their instructional practice in their subject area.63 Whenever content is being learned from text (print, digital, or other), instructional support for the necessary reading comprehension skills should be embedded in instruction.
- Ensure that multiple literacy skills are integrated within curricula and instructional materials, and provide exemplars. State and district leaders play a critical role in setting expectations and providing exemplars for quality curricula and instructional materials. While at times it may be appropriate to teach reading comprehension in isolation, most often, advanced literacy skills—including listening, language, vocabulary, perspective-taking—should be integrated into content area instruction as the skills and strategies necessary to achieve larger learning goals such as how to build new knowledge with texts, reason and debate an issue, or solve a problem with texts.64 It follows, then, that:
- The development of content area instructional units should involve both literacy teachers and content area teachers to ensure that students are supported in the development of reading comprehension skills across the preK-12 continuum.
- Academic language development should be facilitated concurrently with reading comprehension development across the entire span of schooling. This variant of language is the currency of learning environments and should be practiced and developed productively, both in classroom discussions and in written communication. Informal, conversational language use should also be encouraged, but helping students to learn and command an academic register of language use will strengthen their confidence and competence in future learning (and workplace) settings.
- Realistic texts of varying types and formats should be incorporated across the curriculum, including fiction and nonfiction, print, digital, and other to reflect the many sources of information, inspiration, and communication students will encounter as adults. Textbooks and trade books written explicitly for educational use serve a didactic purpose. But the world of literacy beyond the school is much richer and more varied than the controlled and closely edited texts prepared by educational publishers and test companies. The internet better represents the richness of literacy the student can expect beyond schools. While access to technology in today's classrooms can be limited by school funding, failure to integrate technology with learning underrepresents the reading comprehension construct and may increase the achievement gap between students from low-income families and their more advantaged peers. In addition, technology affords access to information and an opportunity to exercise critical thinking skills, and can be used to scaffold and enrich literacy environments.
- Curricula should encourage language-rich environments with discussion and debate of the ideas and content found in texts as a primary pedagogical vehicle for increasing comprehension. The social context of literacy use and practice should be seamless and synonymous with the practice of learning to comprehend individual text sources. Strong readers often self-explain in their own heads when constructing and reasoning about texts. However, this internal dialogue should be modeled externally for less skilled readers and scaffolded through social interactions.
- As indicated previously, instruction should directly address the development of thinking and reasoning skills concurrent with comprehension development across the lifetime. This development should occur even before students can read independently. Instruction in the content areas, including English classes, needs to include the specific thinking and reasoning strategies required in that discipline. The act of reasoning or thinking about text itself is a knowledge-building activity. In this sense, it both supports comprehension and is comprehension.65
- The context and setting of literacy practices within instructional units should be purposeful, goal directed, and engaging. Students should not read to simply answer basic reading questions, but to solve problems or make decisions as they engage in rich literacy activities. Comprehension is a means to an end, not merely an end in itself.
- Ensure that all content area instruction includes "reading to learn." "Reading to learn" requires the student to apply multiple comprehension skills, including:
- extracting and evaluating information from texts;
- identifying the big, conceptual ideas across texts and not merely the "main ideas" of individual texts;
- reasoning about the content of text and how it relates to or contradicts one's prior knowledge;
- making inferences that connect text ideas coherently, as well as relating them to one's prior knowledge; and
- generalizing one's knowledge beyond the text and topic in question.
Some of the strategies employed vary across disciplines. Therefore, a disciplinary approach to reading should be used by districts, particularly at the middle and high school levels, so that all students learn to "read to learn" in each discipline.
- Ensure that the texts used within instruction increase in reading level, complexity, and length across the grades. In order to be prepared for the reading demands of postsecondary education, students should be called upon to develop fluency and stamina with language as represented in print sources through the inclusion of texts of increasing length, complexity, and volume that are commensurate to (and sometimes in advance of) their grade levels. Text complexity metrics are far from an exact science and are mediated by students' prior knowledge and interest in the topics of texts.66 Students should never be restricted from attempting to read at any text level because of their current ability. Complex texts can help students to learn how to make inferences, apply strategies, and build stamina in the face of challenges. Simple texts of high interest can help students reinforce fluency, acquire new knowledge, and support engagement and enjoyment in reading.
- Monitor and develop policies to support dispositions so students see engagement in print reading as natural and necessary to their personal and social development. This goes beyond appreciation for literature or a joy of reading. Increasingly, easy access to social media, smartphones, and streaming video are competing with extended print reading for students' time and engagement. For many purposes, these alternate media (which often include some limited print reading requirements and navigation skills) are sufficient for learning or solving a problem (e.g., how-to videos). States and districts need to show leadership and devise creative strategies that schools and teachers can use to engage students. Each RfU team addressed motivation and engagement. No simple answer to engaging students emerged, though a strong focus on goal-directed, relevant learning activities, engaging students in big conceptual questions in social learning environments, might be sufficient in the near term for most classrooms.67 However, the long-term problem of students disengaging from reading and learning in their school careers is a problem that needs to be addressed.
The implications of the research conducted under the RfU initiative are quite significant for teacher preparation programs and for state and district professional development programs. First and foremost, it is now clear that reading proficiency should be the responsibility of every teacher (not just reading and English/language arts teachers), as well as every educational professional and school administrator. This includes content area teachers, whose aspirations should not only be about building students' content understanding but also to develop the specific skills used within that discipline to learn from and critically evaluate content.
During the RfU project, professional development took on many forms, from assistance with implementing interventions, to co-development of materials and programs, to knowledge transfer on the latest best practices. District leaders may want to require principals to identify faculty and staff with primary responsibility for reading literacy development and organize annual schoolwide reading literacy development plans, with monitoring of progress. A parallel but less intensive plan could be implemented for faculty-wide reading literacy development for all other content area faculty. We recognize that the recommendations here will require substantial planning in order to implement in feasible, resource-reasonable ways. Nonetheless, we see several high priority investments that may be required.
Recommendations for state and district leaders regarding professional development:
- Require that teacher preparation incorporate research-based, discipline-specific reading comprehension training into the pre-service training of all teachers. Also, because the field of cognitive science is rapidly developing, states should require teacher preparation programs to update their curriculum to reflect new advances in comprehension instruction as they are developed.
- Require in-service teachers who are not proficient in the instruction of discipline-specific reading comprehension skills to engage in professional development and then incorporate it into instruction. Comprehension instruction should reflect the expectations of the discipline. Consequently, content-area teachers will need expert knowledge about how to teach reading for understanding in their discipline. For instance, teachers and students should read history texts in a manner that is consistent with disciplinary expectations (e.g., distinguish primary from secondary sources); this is different from how they should read a science text.
The RfU projects identified several insights into key elements of such professional development offerings.
- Receptive and productive development of academic language should be a priority, especially with underserved groups whose language experiences outside of school may diverge from the academic language use expected in school.
- Knowledge building across topic and subject areas using rich oral language, vocabulary, as well as visual-graphic materials should continue to be a priority, with attention being given to how this knowledge can be integrated with text reading and comprehension.
- Social interactions and communication that foster perspective taking and multiple points of view should also be developed as prerequisites to advanced comprehension skills. Students need to be agents of their own learning, but also collaborators in a socially constructed environment.
- The foundations for evaluating the credibility and integration of multiple sources (whether text, oral, or visual) should also be introduced as preparation for more advanced multiple source reading comprehension.
- Teachers should keep up to date on the forms, genres, devices, and uses of digital technologies as they represent the reading literacy world that they are preparing students to enter. While the type and prevalence of devices and displays (e.g., laptops, tablets, smartphones, smart boards), communication platforms (e.g., email, blog, Twitter, Snapchat), and resources (e.g., World Wide Web, Wikipedia) continue to change and expand at a dizzying pace, educators must do their best to prepare children from the earliest ages to be flexible in their approach to learning and adapting to dynamic literacy environment of our age. This requires a mindset for integrating both formal reading and writing contexts with more informal and dynamic digital environments.
- Teachers and educators need to enhance their ability to create language-rich environments and discussions that foster language development and listening comprehension as well as reading comprehension. In the early grades, language development and listening comprehension instruction should be provided even before students have learned or mastered reading of printed text.
In addition to, and because of, the need for a systematic effort to open successive gates to opportunity, it is essential to adopt a systemic approach that encourages, facilitates, and takes advantage of all relevant resources, funders, and providers. Effective solution strategies and implementations cannot simply depend on the efforts of, and utilize resources within, a single organization or system. ... The enormity of the challenges our nation faces demands an 'all hands on deck' philosophy and commitment. ...
In a long-term effort, no single, fixed initiative can carry the burden of change. Rather, in view of the complexity of the challenge and a constantly shifting environment, a strategy of continuous improvement must be implemented. This approach, in which incremental modifications are made over time, involves establishing networks to support collaboration and providing a forum so that providers, system participants, researchers, and funders can learn from each other, as well as from the mistakes and failures that most certainly will occur. Those modifications should be informed both by evidence from a particular site and by evidence from other, similar sites.
Kirsch et al., Choosing Our Future
Improving reading literacy involves sustained work at both the classroom level and the larger, community, state, and national level. It also requires research to devise and test new approaches to accelerate learning. If continuous improvement in reading literacy is to ever be achieved, there have to be mechanisms that allow solutions to be tested within classrooms, evaluated, implemented more broadly, and sustained over time. The RfU research teams worked with schools that had a need to enhance reading outcomes for their student populations. On the whole, each of the RfU projects brought technical expertise, ready-to-use professional development, curriculum and instructional content, evaluation tools, and monetary resources to schools. Each team did its best to be accommodating to the curriculum and logistical needs of participating schools and districts. Yet each faced multiple barriers to implementing its instructional programs, making trade-offs that sometimes compromised the evaluation of the efficacy of the programs. As of the writing of this report, teams were less than optimistic that schools would be able to sustain the instructional programs they put in place beyond the duration of the projects.
Imagine that we had solid evidence that the instructional programs implemented by the research teams would be effective. What would it take for them to become instantiated and institutionalized in the schools? It is toward this aim that we offer the following recommendations for district leaders, as well as the state leaders who advise and support them.
Recommendations for state and district leaders regarding implementation:
- Ensure that the rules, regulations, and policies that govern school organization and schedules can accommodate implementation and sustainability of new programs designed to enhance student achievement. The role of district leadership in transforming the teaching of language and literacy will be essential to implementing and sustaining change. Coverage of the curriculum is important, but content cannot really be learned or mastered absent advanced comprehension skills needed to sustain and enhance disciplinary learning. Time should be set aside for educational staff to explore new approaches to student learning and professional development.
- Integrate the expectation of innovation and change into policies and plans. While structure and routine schedules are important for the success of any institution, a mindset for innovation and improvement among districts is important for keeping pace with a dynamic learning environment.
- Monitor how well change is sustained over the long term. Reading is a complex skill that develops over time; changes are gradual, with slow growth over time. Educators should explore the use of new measures that can capture smaller, more realistic changes in student growth and to determine whether, over time, student learning has improved. If so, it is equally important for districts to continue monitoring to ensure that implementation does not decline.
- Encourage curriculum design research that collaboratively engages educators with researchers and provides time to explore and implement new approaches for improving reading. Implementing and sustaining innovation in research in schools will require a different relationship between educators and research. The approach needs to take into account not only the logistics of implementing a study, but also factors impacting the likelihood of instantiating a sustainable change.
36 John Sabatini and Tenaha O'Reilly, "Rationale for a New Generation of Reading Comprehension Assessments," in Unraveling the Behavioral, Neurobiological, and Genetic Components of Reading Comprehension, eds. Brett Miller, Laurie Cutting, and Peggy McCardle (Baltimore, MD: Brookes Publishing, 2013), 100-111.
37 John Sabatini, Kelly Bruce, Jonathan Steinberg, and Jonathan Weeks, SARA Reading Components Tests, RISE Forms: Technical Adequacy and Test Design, ETS Research Report Series (Princeton, NJ: Educational Testing Service, 2015): 1-20, doi:10.1002/ets2.12076.
38 LaRusso et al., "Contributions of Academic Language," 201-222.
39Donald Shankweiler, Eric Lundquist, Lois G. Dreyer, and Cheryl C. Dickinson, "Reading and Spelling Difficulties in High School Students: Causes and Consequences," Reading and Writing 8, no. 3 (1996): 267-294, doi:10.1007/BF00420279; Timothy Rasinski, Nancy D. Padak, Christine A. McKeon, Lori G. Wilfong, Julie A. Friedauer, and Patricia Heim, "Is Reading Fluency a Key for Successful High School Reading?" Journal of Adolescent & Adult Literacy 49, no. 1 (2005): 22-27, doi:10.1598/JAAL.49.1.3
40 Gordon Commission, To Assess, to Teach, to Learn: A Vision for the Future of Assessment (Princeton, NJ: Educational Testing Service, 2013).
41Randy Bennett, "Cognitively Based Assessment of, for, and as Learning (CBAL): A Preliminary Theory of Action for Summative and Formative Assessment," Measurement: Interdisciplinary Research and Perspectives 8, no. 2-3 (2010): 70-91.
42 John Sabatini, Laura K. Halderman, Tenaha O'Reilly, and Jonathan P. Weeks, "Assessing Comprehension in Kindergarten through Third Grade," Topics in Language Disorders 36, no. 4 (2016): 334-355, doi:10.1097/TLD.0000000000000104; John Sabatini., Tenaha O'Reilly, Laura K. Halderman, and Kelly Bruce, "Broadening the Scope of Reading Comprehension Using Scenario-Based Assessments: Preliminary Findings and Challenges," L'Année Psychologique 114, no. 4 (2014): 693-723, doi:10.4074/S0003503314004059; Tenaha O'Reilly, Jonathan Weeks, John Sabatini, Laura Halderman, and Jonathan Steinberg, "Designing Reading Comprehension Assessments for Reading Interventions: How a Theoretically Motivated Assessment Can Serve as an Outcome Measure," Educational Psychology Review 26, no. 3 (2014): 403-424, doi:10.1007/s10648-014-9269-z.
43 John Sabatini et al., "Assessing Comprehension," 334-355.
44 John Sabatini, Tenaha O'Reilly, and Paul Deane, "Preliminary Reading Literacy Assessment Framework: Foundation and Rationale for Assessment and System Design," ETS Research Report Series (Princeton, NJ: Educational Testing Service, 2013), doi:10.1002/j.2333-8504.2013.tb02337.x; Tenaha O'Reilly and Kathleen M. Sheehan, "Cognitively Based Assessment of, for, and as Learning: A Framework for Assessing Reading Competency," ETS Research Report Series (Princeton, NJ: Educational Testing Service, 2009), doi:10.1002/j.2333-8504.2009.tb02183.x.
45Danielle McNamara, Manuel de Vega, and Tenaha O'Reilly. "Comprehension Skill, Inference Making, and the Role of Knowledge," in Higher Level Language Processes in the Brain: Inference and Comprehension Processes, eds. Franz Schmalhofer and Charles Perfetti (Mahwah, NJ: Erlbaum, 2007), 233-253.
46 M. Anne Britt, Jean-Francois Rouet, and Amanda Durik, Literacy beyond Text Comprehension: A Theory of Purposeful Reading (New York: Routledge, 2017).
47 Zuowei Wang, John Sabatini, Tenaha O'Reilly, and Gary Feng, "How Individual Differences Interact With Task Demands in Text Processing," Scientific Studies of Reading 21, no. 2 (2017): 165-178, doi:10.1080/10888438.2016.1276184.
48 Tenaha O'Reilly, John Sabatini, and Zuowei Wang, "Using Scenario-Based Assessments to Measure Deep Learning," in Deep Learning: Multi-Disciplinary Approaches, eds. Keith Millis and Joe Magliano (New York: Routledge Taylor & Francis, in press).
49 Robert Mislevy and Geneva Haertel, "Implications of Evidence‐Centered Design for Educational Testing," Educational Measurement: Issues and Practice 25, no. 4 (2006): 6-20, doi:10.1111/j.1745-3992.2006.00075.x.
50 William Nagy, Virginia Berninger, and Robert Abbott, "Contributions of Morphology beyond Phonology to Literacy Outcomes of Upper Elementary and Middle-School Students," Journal of Educational Psychology 98, no. 1 (2006): 134, doi:10.1037/0022-06220.127.116.11
51Tenaha O'Reilly and John Sabatini, "Reading for Understanding: How Performance Moderators and Scenarios Impact Assessment Design," ETS Research Report Series (Princeton, NJ: Educational Testing Service, 2013); Amy Shapiro, "How Including Prior Knowledge as a Subject Variable May Change Outcomes of Learning Research," American Educational Research Journal 41, no. 1 (2004): 159-189, doi:10.1002/j.2333-8504.2013.tb02338.x.
52 Henry Braun, Irwin Kirsch, and Kentaro Yamamoto. "An Empirical Study to Examine Whether Monetary Incentives Improve 12th Grade NAEP Reading Performance," Teachers College Record 113 (2011): 2309-2344.
53Goldman, "Adolescent Literacy," 89-116; Goldman et al., "Disciplinary Literacies," 219-246.
54LaRusso et al., "Contributions of Academic Language," 201-222; Miriam Metzger, "Making Sense of Credibility on the Web: Models for Evaluating Online Information and Recommendations for Future Research," Journal of the Association for Information Science and Technology 58, no. 13 (2007): 2078-2091, doi:10.1002/asi.20672; Britt, Rouet, and Durik, Literacy beyond Text Comprehension.
55 Coiro, "Rethinking Online Reading Assessment," 59-63.
56 The call for proposals for RfU initiative was released before the Common Core State Standards.
57Carol McDonald Connor, Ralph Radach, Christian Vorstius, Stephanie L. Day, Leigh McLean, and Frederick J. Morrison, "Individual Differences in Fifth Graders' Literacy and Academic Language Predict Comprehension Monitoring Development: An Eye-Movement Study," Scientific Studies of Reading 19, no. 2 (2015): 114-134, doi:10.1080/10888438.2014.943905; Kenn Apel and Emily Diehm, "Morphological Awareness Intervention with Kindergarteners and First and Second Grade Students from Low SES Homes: A Small Efficacy Study," Journal of Learning Disabilities 47, no. 1 (2014): 65-75, doi:10.1177/0022219413509964; Jamie Quinn, Richard K. Wagner, Yaacov Petscher, and Danielle Lopez, "Developmental Relations between Vocabulary Knowledge and Reading Comprehension: A Latent Change Score Modeling Study," Child Development 86, no. 1 (2015): 159-175, doi:0.1111/cdev.12292; Language and Reading Research Consortium, Ann Arthur, and Dawn Davis, "A Pilot Study of the Impact of Double-Dose Robust Vocabulary Instruction on Children's Vocabulary Growth," Journal of Research on Educational Effectiveness 9, no. 2 (2016): 173-200, doi:10.1080/19345747.2015.1126875.
58Nathan H. Clemens, Deborah Simmons, Leslie E. Simmons, Huan Wang, and Oi-man Kwok, "The Prevalence of Reading Fluency and Vocabulary Difficulties among Adolescents Struggling With Reading Comprehension," Journal of Psychoeducational Assessment 35, no. 8 (2016), 785-798, doi:10.1177/0734282916662120; Carolyn A. Denton, Mischa Enos, Mary J. York, David J. Francis, Marcia A. Barnes, Paulina A. Kulesz, Jack M. Fletcher, and Suzanne Carter, "Text‐ Processing Differences in Adolescent Adequate and Poor Comprehenders Reading Accessible and Challenging Narrative and Informational Text," Reading Research Quarterly 50, no. 4 (2015): 393-416, doi:10.1002/rrq.105; Leslie Duhaylongsod, Catherine E. Snow, Robert L. Selman, and Suzanne M. Donovan, "Toward Disciplinary Literacy: Dilemmas and Challenges in Designing History Curriculum to Support Middle School Students," Harvard Educational Review 85, no. 4 (2015): 587-608, doi:10.17763/0017-8055.85.4.587; Sarah Levine and William Horton, "Helping High School Students Read Like Experts: Affective Evaluation, Salience, and Literary Interpretation," Cognition and Instruction 33, no. 2 (2015): 125-153.
59LaRusso et al., "Contributions of Academic Language," 201-222; Stephanie M. Jones, James Kim, Maria LaRusso, Ha Yeon Kim, Paola Uccelli, Sophie Barnes, Suzanne Donovan, and Catherine Snow, "Experimental Effects of Word Generation on Vocabulary, Academic Language, and Perspective Taking in High Poverty Middle Schools," Society for Research on Educational Effectiveness (2016); Paola Uccelli, Emily Phillips Galloway, Christopher D. Barr, Alejandra Meneses, and Christina L. Dobbs, "Beyond Vocabulary: Core Academic Language Skills (CALS) that Support Text Comprehension," Reading Research Quarterly 50, no. 3 (2015): 261-359, http://nrs.harvard.edu/urn-3:HUL.InstRepos:17116368.
60 Barbara Foorman and Jeanne Wanzek, "Classroom Reading Instruction for All Students," in Handbook of Response to Intervention, eds. Shane Jimerson, Matthew Burns, and Amanda VanDerHeyden (Springer, 2016): 235-252; Elizabeth Swanson, Jeanne Wanzek, Lisa McCulley, Stephanie Stillman-Spisak, Sharon Vaughn, Deborah Simmons, Melissa Fogarty, and Angela Hairrell, "Literacy and Text Reading in Middle and High School Social Studies and English Language Arts Classrooms," Reading & Writing Quarterly 32, no. 3 (2016): 199-222, doi:10.1080/10573569.2014.910718.
61Language and Reading Research Consortium et al., "A Pilot Study," 173-200; Betty Hart and Todd Risley, Meaningful Differences in the Everyday Experience of Young American Children (Baltimore, MD: Paul H Brookes Publishing, 1995).
62Clemens et al., "The Prevalence of Reading Fluency"; Marcia Barnes, Karla K. Stuebing, Jack M. Fletcher, Amy E. Barth, and David J. Francis, "Cognitive Difficulties in Struggling Comprehenders and Their Relation to Reading Comprehension: A Comparison of Group Selection and Regression-Based Models," Journal of Research on Educational Effectiveness 9, no. 2 (2016): 153-172, doi:10.1080/19345747.2015.1111482; Amy Barth, Marcia Barnes, David Francis, Sharon Vaughn, and Mary York, "Inferential Processing Among Adequate and Struggling Adolescent Comprehenders and Relations to Reading Comprehension," Reading and Writing 28, no. 5 (2015): 587-609, doi:10.1007/s11145-014-9540-1.
63 Susan Goldman, Cynthia Greenleaf, Mariya Yukhymenko-Lescroart, Willard Brown, Monica Ko, Julia Emig, MariAnne George, Patricia Wallace, Dylan Blum, M. Anne Britt, and Project READI, "Explanatory Modeling in Science through Text-Based Investigation: Testing the Efficacy of the READI Intervention Approach," Project READI Technical Report No. 27 (2016); Marcia A. Barnes, Yusra Ahmed, Amy Barth, and David J. Francis, "The Relation of Knowledge-Text Integration Processes and Reading Comprehension in 7th- to 12th-Grade Students," Scientific Studies of Reading 19, no. 4 (2015): 253-272.
64 As an analogy, think of how science teachers consider it their responsibility and mission to teach research skills appropriate to the science discipline. In chemistry, this includes using beakers, flasks, and burners to mix chemicals or prepare solutions; in biology this might include dissections or growing cultures, etc. Now think of reading in the disciplines as another research skill needed to learn chemistry or science knowledge and reasoning from text sources about those sciences.
65 It is often overlooked that reasoning generates knowledge. It is not just a process applied to text content after knowledge is acquired. When one reasons about or reflects on a text, or compares and contrasts ideas on a topic, one generate inferences or reorganizes what one knows about the topic. This is new knowledge that was perhaps not ever represented explicitly in the source texts.
66 Mienke Droop and Ludo Verhoeven, "Background Knowledge, Linguistic Complexity, and Second-Language Reading Comprehension," Journal of Literacy Research 30, no. 2 (1998): 253-271, doi:10.1080/10862969809547998; Elfrieda Hiebert and Heidi Anne Mesmer, "Upping the Ante of Text Complexity in the Common Core State Standards: Examining its Potential Impact on Young Readers," Educational Researcher 42, no. 1 (2013): 44-51, doi:10.3102/0013189X12459802; David Gamson, Xiaofei Lu, and Sarah Anne Eckert, "Challenging the Research Base of the Common Core State Standards: A Historical Reanalysis of Text Complexity," Educational Researcher 42, no. 7 (2013): 381-391, doi:10.3102/0013189X13505684.
67Language and Reading Research Consortium, "Use of the Curriculum Research Framework (CRF) for Developing A Reading-Comprehension Curricular Supplement for the Primary Grades," Elementary School Journal 116, no. 3 (2016): 43-70; Amy Pratt and Jessica Logan, "Improving Language-Focused Comprehension Instruction in Primary-Grade Classrooms: Impacts of the Let's Know! Experimental Curriculum," Educational Psychology Review 26, no. 3 (2014): 357-377; Susan Goldman and James Pellegrino, "Research on Learning and Instruction: Implications for Curriculum, Instruction, and Assessment," Policy Insights from the Behavioral and Brain Sciences 2, no. 1 (2015): 33-41, doi:10.1177/2372732215601866; Duhaylongsod et al., "Toward Disciplinary Literacy," 587-608.