Reading for Understanding: Research Panel Discussion

People in this Video:

T.J. Elliott, VP and Chief Learning Officer, ETS
Elizabeth Albro, Associate Commissioner, National Center for Education Research
Barbara Eason-Watkins, Superintendent, Michigan City Schools in Indiana
Jessica Chambers, Literary Specialist, Chicago Public Schools
Art Graesser, Professor, University of Memphis
Eunice Greer, Senior Research Scientist, National Center for Education Statistics
Don Compton, Associate Director, Florida Center for Reading Research

On-screen: [ETS. CCSSO. Council of Chief State School Officers.].

T.J. Elliott – Good afternoon. This is our research panel and then that would be followed at 2:00 by our policy and practice panel which will also be part of a webcast. I'm T.J. Elliot. I'm Vice President and Chief Learning Officer at ETS and I really feel privileged to have been part of this conference for the last two days. I feel a little bit like I've been part of a cooking program, where we've stirred and we've sliced and we diced and we pureed, and now we're going to see what comes out after being baked out here in these next two panels here. So the desired outcomes of this session are that we reach a synthesis of the key research and learning insights that cut across the entire RFU research initiative. The key takeaways generated in Sli.do had been shared with this panel and the next panel as a starting point for their discussion. Also, this panel is charged with trying to identify perhaps a more concise set of issues to be explored by the practice panel and policy panel which will be session five. These issues we can already see go beyond the RFU projects to discuss how intervention, alignment, assessment, research, in the broader context, needs to be evaluated and shaped for it to address these emerging high leverage needs and to improve impact. And so I'm going to ask each panel member after I introduce them to summarize in three or four minutes one or two of these high leverage research issues that they see. I'm going to ask everyone who's in the audience here to continue using Sli.do and if you're so moved when we get to that portion of the program to step up to one of the microphones here. But first let me introduce our panelists.

So first, Elizabeth Albro is Acting Commissioner of the National Center for Education Research, Institute of Education Sciences.

Elizabeth Albro – I'm the Associate Commissioner.

T.J. Elliott – Oh, there we go. We have an issue already.

Elizabeth Albro – So my boss just walked in he'll be like, "what?"

T.J. Elliott – Maybe you don't know what happened during lunch —at the US Department of Education.

Barbara Eason-Watkins is Superintendent of the Michigan City Schools in Indiana.

Jessica Chambers, Literary Specialist at Chicago Public Schools.

Art Graesser, Professor in the Department of Psychology and the Institute of Intelligent Systems at the University of Memphis and a senior research fellow in the department of education at the University of Oxford.

Eunice Greer, Senior Research Scientist at the National Center for Education Statistics.

Originally, Annemarie Palincsar was going to be here, but stepping in for her because she had some airline troubles, very nicely is Don Compton. Don is the Associate Director at the Florida Center for Reading Research.

So let's give them all a welcome. [clapping]

Elizabeth I'm going to ask you to go first with your three or four minutes.

Elizabeth Albro – Great. Thanks everyone so much for all of your participation and engagement in this event that I have to say, I can hardly believe actually has happened. Karen Douglas and I have the privilege and honor of working with the Reading for Understanding teams from the beginning. I want to just start out and say the ways in which the successes that the teams have made, that you all have been commenting on, are really already evident. One of the things that we talked when we were envisioning this process was that we wanted to have interdisciplinary networks of teachers, researchers, and people on the ground right, not just the teachers, but the whole practice community. Working together to try to think how we tackle this problem of reading comprehension across the developmental spectrum, right. So one of the real challenges when this initiative was originally conceptualized was we'd spent an awful lot of time and investment thinking about how to get kids reading on grade level by third grade. But for those of us in this room we know that that doesn't, reading doesn't end at third grade and that there was an awful lot more that we needed to understand about how comprehension continues through the grades. The first thing is that we were successful in creating these interdisciplinary network of teams of teams. I was trying to think how to do this. So you guys are only getting, in some ways, the surface level of the folks who are behind all of this. So we were successful in doing that. I think that another key component that I've heard talk about over and over again in this meeting, was we really wanted to understand and do a research enterprise around language, and what is the role of language and reading comprehension. So when we started this endeavor there'd been lots of work to suggest that language was critically important in terms of being related to reading comprehension. But we hadn't had systematic efforts in terms of thinking about how that worked together. So I think we've learned a lot about conversation and I think that there are lots of research questions around how do we take what we learned about incorporating language into classrooms and supporting teachers and the sort of classroom structures to help make conversation, and talk, and language a reality across all of the classrooms. So I put that out as one research question to think about is how do we take that language construct, how do we take that conversation, and what do we need to know to support teachers to do that, right. So that is a piece of what we've done.

Sorry, we were given this long list of all the things that came out on Sli.do and I'm looking at my notes and realizing my notes don't actually map onto what I had in Sli.do. I think that the other piece that I wanted to highlight is when we think about what to do next, right. The other message that came clear, and I think Louis Gomez spoke about it this morning, was thinking about the systems that we need to have in place within the education context to support the uptake of everything that we've learned. And that's a research question and also a practice question, right. So we know that systems are important, but I think there's a lot more for us to unpack as we try to think through, how do we take what we as researchers learn and incorporate them into the space of practice. So, the last piece that I'd like to put out there is that IES has been investing and developing researcher practitioner partnerships and relationships. Trying to think about how do we help build the conversation explicitly across the research; and I actually don't like the word practice I feel like it doesn't quite capture what we're talking about, but how do we build those bridges and I just want to invite those of you as you leave the meeting, as you start to think about this, think about ways to think about whether there might be opportunities for you to create explicit partnerships where we can continue this conversation.

T.J. Elliott – Thank you very much.

Elizabeth Albro – You're welcome.

T.J. Elliott – Barbara, you're doing double duty here, you're on this panel and then we're making you go on the next panel as well here. What issues occurred to you?

Barbara Eason-Watkins – Well, what I heard across all the groups, and I think it's very positive, is the fact that there's a lot that we can build upon in terms of teacher support and teacher professional development. But across several of the projects that we learned about today, clearly the lack of leadership support, the lack of understanding among leaderships in supporting the projects came through. And in terms of a problem of practice, how do we support leaders, both school leaders and district leaders, in ensuring that the necessary supports are in place, that decisions aren't made that impact the implementation of a project. What are the routines that need to be established at the school and district level to ensure that they are part of the learning process as well, rather than just allowing a program to come into a building without understanding that this is part of a systemic change that's taking place across the district? From my perspective, if we can move forward and incorporate the leadership aspect in the next level of work from a research perspective, then we would be able to inform, not only these projects, but I think work across, you know, multiple areas that impact schools and districts.

T.J. Elliott – You touched upon a number of things that were raised here. I'm going to hand out for those of you who were here, this is the Sli.do issues that we could take a look at those as we move through the rest of the discussion here. Jessica, what's your take? You come at this from a somewhat different perspective and a really important one, in terms of practice here. What boils up for you out of all of these wonderful insights?

Jessica Chambers – As a teacher until November and now someone working at the level of the district, one of the things that continues to stick out for me in talking about these projects, is the real need to treat teachers as partners in this work. In addition to providing really great curricular materials, we need to support teachers in building their own expertise and deepen their understanding so that they can then take these great ideas that the researchers have and really build on them in tandem with the materials so what we come out with at the end is greater even than what we started with because of those teacher practices in the schools. That necessitates a creation of communities and schools that really privilege ongoing learning and allow a safe space for teachers to practice and fail and practice again and fail again. And that we have administrators and leaders in our districts and in our schools that understand that just like we allow our students to fail on the practices of learning, that we have to allow that same thing for our teachers. Then, to build really strong models for other schools who aren't doing this work of what this looks like when it's done well. What are the systems and the infrastructures and what does it look like in classrooms where this is happening? So I'm left with this question of how we support teachers then, who are doing this work well, and draw upon that expertise to support work in the larger school communities. So really treating teachers as partners, growing their leadership, and then allowing them to support other teachers who might need more help in building their own expertise.

T.J. Elliott – Putting your two points together there in at least my mind is an interesting connection that we say, well you have to fail in order to learn, and if we were going to support teachers who were doing this work well we would support them and giving them that space to do that there but that's more easily said than done from what you're expressing.

Art Graesser …

Art Graesser – Thanks. I want to make three sort of pitches. The first has to do with adults, 18 and beyond. You know they continue learning and have reading needs. 1 in 6 adults don't read at a level that will get them a decent job or be happy in their community. I'm part of a center for the study of adult literacy, one of the IES funded centers, and that's our mission. To try to understand research on struggling adult readers. Many of them, the teachers and tutors at literacy centers, don't have much training and so the professional development is a big challenge.

Now, I want to bring up one thing that hasn't been emphasized here and that's technology. And I think technology can both help us for the adults, the struggling adult readers, but also for K12. And we all know that technology, access to the internet, can start a whole new world and has already started a new world where they're spending many hours on the internet, whether it's games or social media. That's a real part of the world we live in. Now, what we do is develop intelligent tutoring systems, very adaptive systems to the individual students. I'm particularly fond of those that have conversational agents, talking heads, where you can have a tutor agent and a student agent. We have what they call trialogs, where two agents interact with a human, and a few people like my colleague Danielle McNamara, or Anne Britt--they have developed some of these systems. I think they can be very useful in scaling up what we do because it's on the web, it can be shared by hundreds, thousands, ideally maybe even millions to be used.

That leads to the third point I want to make. We've developed about 35 of these modules on different comprehension skills and they're all used and they have learning gains, but we think these can be used in professional development. This is how the vision works; imagine if teachers critique these lessons. They're critics, you pay them to be critics, but sneakily, actually, you're training them, so they learn strategies by critiquing these. And then what we do is use those data to actually modify our systems. So the teachers will have, and it's not just a handful of teachers, it may be thousands of teachers have critiqued each of these 35 modules, so over time you have much improved modules and of course we have authoring tools where they can build their own, but of course that's very hard to do. So the idea is the teachers are critics, helped co-develop these systems, and then the systems can help the struggling adults, but this could go to K12 also, and so you have a system there of facilitating this partnership.

Just one last footnote. You know we are in the era of big data, where you can do machine learning on large data sets, by large we're thinking in terms of millions. I think you can come up with very adaptive systems if you do that. So I think this is an exciting point in history and I'm glad to be part of it.

T.J. Elliott – You've given the next panel a lot to think about there as your colleagues have. Continuing down the row here...Eunice, what really stuck out for you?

Eunice Greer – Well, first I want to thank Liz and her colleagues for inviting me to join you today. I'm sort of a visitor so this is my first sort of few days with RFU and I work on NAEP and so NAEP is charged with taking very good care of a trend line that began in 1992, and that has followed our nation's children and their progress in reading for several decades. So we're charged with developing an assessment that assesses reading, math, a variety of subjects and honors and respects the way the subjects are being taught. So it's very important that we listen and we come to conversations like this and we learn from you and I have learned so much in the last two days. I am so grateful for the conversations that I have heard regarding the role of language and reading. You've given me a whole lot of work to go back and to follow up on. It's great, it's what makes our work exciting. Our assessments are guided by frameworks, many of you probably know that. I know several of you worked on the last framework and we get a new framework roughly every ten years. The frameworks are owned by the National Assessment Governing Board. They are our marching orders for our assessments. So the last one, Terry Salinger led the development at AIR for the last framework and it was finished in 2009. So that work began what, Terry, in 2006?

Terry Salinger [off screen--in the audience] - No it was actually earlier, in 2003.

Eunice Greer – So we are just about due for a new framework. So you've created for us, really a perfect storm. You've thrown down the gauntlet and said, "language plays a much more important role in reading and literacy than we've acknowledged before. So NAEP, what are you going to do about that?" So that's my big research question, how do I take that back [the research findings] and [we're] on the edge of hopefully a new framework that reflects more contemporary visions of reading, what's the role of language going forward for NAEP? How should NAEP address this really exciting role, that we know now from your work, language plays in understanding how students read. So that's one piece.

Another piece, listening to John and Tenaha and the work that they're doing around scenario based assessments, NAEP is verging into scenario based assessments as well as we transfer into technology based assessments. But again, we're somewhat limited by the language that we have in a framework that's almost 15 years old. So as we get a new framework, how do we learn from work on things like GISA and RISE and Barbara Foorman's work on the Florida assessment that guides the way we use technology to reflect a dynamic, problem-focused, purpose-focused assessment of reading, that looks much more like what you're telling me you're working towards, making sure all students have an opportunity to learn and to practice in classrooms.

You've given us those two pieces, and there's one more piece that Art referenced, and that is computer based assessment gives you just this phenomenal wealth of data. We have all this data we can collect about how does a student interact with an assessment. Where do they go? How long do they spend there? Do they go back? Do they change it? What do they skip? What do they ignore? What do they value? All of that is a window into making reading assessment a little bit less of an indirect measure of reading. We're going to get more information, it becomes more closely a measure of what kids are actually doing and what's happening as kids are thinking and comprehending. So we want to learn to use what we're calling that process data. So that when we come back to you to report what we've learned on these assessments, they really are more useful for you. They give you better information, they give you more careful, thoughtful, detailed information about how kids are reading. So we have a lot of work to do and I really want to thank all of you for sort of helping us narrow our focus and giving us a clearer charge.

T.J. Elliott – I think that's got to be very heartening for the people who did the research over the 5-year period, to hear that you have a very specific use for these things coming out of here. Especially some things that challenge some of the assumptions that may have been there in earlier frameworks.

Eunice Greer – We're very excited. We love getting information like this, so thank you very, very much.

T.J. Elliott – Don Compton thanks again for stepping in at the last moment to be a part of this panel here. I know you were also facilitating a lot of discussions here during the two days. What stands out for you?

Don Compton - I'm going to first channel Annemarie and I really want to know where her notes are because I could use them right now. First, I'd like to thank IES for the start to a research agenda. Just because they made this major investment, doesn't mean we're done. I think the research that has been presented here is very exciting, it's very promising, it's very diverse. I think that none of the PIs and team members would say they're done, they understand it all. I noticed in the research sessions toward the end of the day that we moved away from research questions and implementation questions. While those are important, we still have a research mission, we still need basic science funding for moving this work forward. We need to continue the translational work that all of these people have started. We need to then look into the mechanisms for how to move this effectively into the classroom, in the schools. I don't think we're there yet. I mean, I think the analogy we're trying to send a person to the moon and maybe you did that, it's time to go to the stars.

So, my take on the research is we're understanding a lot of the underlying processes that are important to comprehension. We're starting to get a handle on whether we can move those processes. I think we still have a lot of questions in terms of how moving those processes changes comprehension. What's the timeline we should expect? What are reasonable effect sizes? How do we translate those into effects on NAEP and those kinds of things? So I want to make the plug for a continuation of this work, a continuation of commitment by IES and other funders to continue this type of work and not use it as, "oh we've done this." Let's figure out how to get into schools, but let's start that while we continue to learn more about the basic processes of comprehension. It's a terribly complicated endeavor and I think reading comprehension and skilled writing are the areas that we need to know a lot more about.

T.J. Elliott – So that takes us through our initial statements here and now I want to turn to some of the questions. I want to remind folks that the Sli.do app is open and we've got one question there, that I will come to in a moment. I have the advantage of not being part of this field, my field is adult learning, as a Chief Learning Officer in a corporation, but when I heard a lot of things that sounded as if I could be hearing them in my own work place, where people say there's not enough time or you're really having to fight for headspace, and I heard that not just on the implementation side but also how that affected research. You put out a number of modules and only a certain number of schools were able to do that, and I wondered what would be the answer on the research side there of how you deal with the time and headspace issues there as you move forward there. Anyone who feels so moved, please just start in.

Elizabeth Albro – We're looking at Art because we feel he may know the answer.

Art Graesser – When you say headspace, you mean the amount of time to be devoted to …

T.J. Elliott – Well, what specific thing stood out for me this morning, and I hope I'm getting this correctly, but Catherine Snow was talking about the fact that when you took a look at the number of modules that were available, a certain number of teachers used 1/3 of the modules. A number of teachers used over 50% of the modules, and that had an effect.

Art Graesser – Yes, at this phase, what we're doing is putting the auto-tutor modules in real literacy centers to see how teachers are using them. These are primarily tutors. We believe a lot in the one on one. So a big question in our mind is which modules do they choose? How do they use them? Did the students actually use them? And of course what we'd like to do is to have these out in dozens and hundreds of literacy centers and as a first step see how they're used. Then, that will help answer the question of the usage. Unless you do this on hundreds, I don't think you'll have a good enough perspective on scaling it up. A lot of these great systems and interventions that have been created, they're really with a small number of people. I think to scale up, it's got to be hundreds.

T.J. Elliott – Anyone else on that question?

Don Compton – I think we heard from the researchers how difficult it was to get time for their interventions in a very busy school day and whatever they put in they have to take out something. I don't think we know what the proper mix is, I don't know that we know to add this, take this out. So I think the schools are under the very difficult task of trying to educate all children, live up to certain accountability standards, and they want to participate in research, but they also feel like, boy, I can't throw out the pieces that I'm going to be evaluated on. I think there is a real difficult give and take that has to happen and somehow there has to be a system where research and practice are allowed to co-mingle in a way that's beneficial for both sides. I don't think we're quite there yet, but I think these types of projects I think benefited the schools in ways that maybe we can build on them.Maybe you guys can talk about that from a school system standpoint …

Jessica Chambers – I think in hearing you talk, one of the things that sticks out for me is this need for a stronger relationship between districts and the practice side and the research side, because I think that in classrooms and in schools, we spend time on what we value. So if districts had stronger relationships and better ideas about what really should be valued--because it's what works--that would help the administrators and the teachers to spend time on that rather than this kind of Christmas tree approach of everything that we can put together. That districts would drive best practices in this way and that would--I think--that would make a difference in classrooms.

Barbara Eason-Watkins – I would also just add that the role of preservice programs came up in our discussions, and in particular in projects that were working with the grade span of 7-12, in that, in some cases of those projects were forced to spend more time teaching content to teachers that they felt probably could've been addressed in their preservice program. The second issue for the content teachers in those grade levels was that reading was critical, reading comprehension within the discipline, but those teachers had left a preservice program believing that reading was a separate content area. So, how do we then better realign the preservice programs so that they're really serving the needs of, you know, the implementation or the practices that are taking place at the school level?

T.J. Elliott – Eunice, I've got one question here that I think you could handle right away . . . I hope. NAEP just released technology and engineering literacy results. Would reading literacy be next?

Eunice Greer – Would reading literacy be next? I think we need to think about the language that we use. We assess reading on NAEP. The term literacy has evolved and is a bigger piece than reading, the way we used to talk about reading. So I think in moving forward and looking at a new framework we need to look at the language very carefully that we use to describe what it is we're measuring and to define what it is that we really need to be measuring, given what's happening in the context of schools.

T.J. Elliott – The top question here right now on Sli.do reads like this: "Language processes like inference generation continue to develop through high school. Research is needed in how to support language in adolescence." Sort of an invitation for comments about that one way or the other from our panelists here. Who would like to address that?

Elizabeth Albro – I'll just jump in. In terms of thinking about ways to support research in the areas of adolescent reading. So, before "Reading for Understanding" we actually supported a program of research entitled "Interventions for Struggling Adolescent and Adult Readers and Writers." I would just like to note we did not get as many applications as you might think for that area. So that there's a need for the research community to sort of step up and think about the degree to which the grades 7-12 span is a place where we really need to think about reading, writing, and language development embedded in the content knowledge, in terms of the content areas the kids are learning. I think that's really a challenge for the researchers to think about how do we take what we have learned from RFU and what we know about what kids are trying to learn, and think about how we build programs of research around that?

I think another comment that I noticed that really came out is that we also have some two additional populations in our adolescent groups, who we have not spent as much time talking about in terms of RFU than perhaps we needed to. It is just a function of what we were focused on. One has to do with thinking about the struggling readers. So I can't remember what panel I was in earlier and I think it was Carolyn Denton who said this, but reminded us that there are seventh to twelfth graders who still struggle with some of these foundational decoding skills and we often don't think about that. I think we still have a lot of work to do to try to think about how do we actually tackle that problem? This goes to Art's question about adult learners. I think the other piece is about English Learners and I just wanted to share a comment that one of you all made to me when we were talking was that when we think about English Learners we think a lot about the bilingual English-Spanish problem. But that many of you in school districts are actually struggling with how do you provide reading, writing, and language services to many kids who are coming to school from many different language backgrounds? I think that's a huge area of research that we have not even begun to tackle.

T.J. Elliott – Alright you wanted to add a point.

Art Graesser – Yeah, I wanted to . . . When we were working on the adult literacy, I reviewed a lot of the assessments on inferences. I built q-matrices on virtually all of the tests. I found particular tests focused on a narrow band of inferences. I'm particularly excited though in with the GISA test, because that seemed to cover many types. Instead of putting all your eggs on one type of inference, whether it's pronoun references or whether it's causal inferences, the GISA was far superior in covering a broader band of inferences. I think it's exciting that because of these centers that we actually have a better inference foundation to get at that.

Don Compton – I'll make a personal plea too that won't surprise Kate, but we sort of forget we teach comprehension, often times particularly as kids get older, as a set of strategies that are useful, but we forget about the knowledge domain, or the knowledge--general knowledge--kids need to interact with texts. We forget about that complex interaction between personal knowledge and the ability to build a representational text. We have big gaps in terms of kid's knowledge as they move through school, and we need to think about how that influences language processing such as inferencing and all sorts of other processes, working memory capacity and all those. We kind of forget about sort of the knowledge basis of reading and so I'd like to put a plug that, you know, we should think about curriculum, we should think about what we're building over time in terms of knowledge, then also what are the demands from an inference, from a memory, those types of demands and how those vary by child and by what text the child is reading. It's a very complicated system and the more we dumb it down the less we'll understand it, I think.

T.J. Elliott – Watching Sli.do and the questions here is a little bit like my experience the last two nights trying to watch the New York Mets against the Nationals on my computer, it's up, it's down. Currently, our number one question asks, "why don't we have a model of teacher preparation that is more like preparation of doctors?"

Don Compton – Money.

T.J. Elliott – That just proves what a good question it was.

Art Graesser – Well, I don't know a lot about preparations of doctors, but from what I hear it varies all over the place. From teachers in medical schools who deliver fifty facts per hour in a lecture and that's what they're paid to do, all the way to problem-based curriculum, where they just work with cases as a group. They try to solve cases, so as a consequence of four years or whatever, they might've encountered 200 different particular cases. To me, I don't know what exactly it is about the pre-training of doctors that's being referred to, but it seems to cover a big gamut.

T.J. Elliott – I had this experience, my son-in-law is a second year resident, so I knew him all during his medical school training and I was surprised at how few classes he actually attended and how much of his formal instruction was by video tapes that he watched. As you say, it's all over the place. Did you want to add something?

Jessica Chambers – I do think there is something that the medical field does well in that residency program that's practice of apprenticing teachers, practice of apprenticing doctors by digging into the practice and having a model of what good practice in the medical field looks like. Having someone there to bounce ideas off and an apprentice, from a novice to someone who's ready to be a doctor themselves. And I think there are some teacher preparation programs that do that, but I think as teachers and people who prepare teachers, we could do a better job of apprenticing teachers from novice to expert by providing those models of support, while they're in the classroom doing the work.

T.J. Elliott – I want to go to the question that was asked first but now has risen up on board in terms of votes as well here and I wanted to save it towards the summative section of this panel discussion here and someone asked "what do you see is the main contribution of this effort?" And certainly there are a lot of different choices to pick from in that there. What I'd like to do is sort of run through each of the panelists here in turn, and maybe Don I'll start at your end this time here. Sort of the "what and why" you think something is the main contribution of the RFU initiative?

Don Compton – Ok, fair enough. I've been involved with the RFU work since probably its third or fourth year out and I think one of the biggest contributions that the teams have made is their honesty in how difficult it is to move the distributions substantially in terms of reading comprehension and all of the components that go into the amount of thought, the amount of sheer sweat equity that has to go into that. I think it was, I know people say there is no silver bullet, but there is no silver bullet. It doesn't happen in a year, it doesn't happen from a program it happens from dedication moving all the way through early language, to breaking the code, to reading, using reading to then learn content. None of these happen quickly. None are easy and they all rely on really committed people to do it. I'm sure people are like well, duh, but you know that's what it takes. But the other realization is we can do it better and we need to keep studying how to do it better and people need to continue to work at doing it better. Our job will never be done here. I think the effort that these teams put in and the thought gives us a great kind of building block to move forward. I applaud that for what they've done.

T.J. Elliott – That's a great start.

Eunice?

Eunice Greer – This is a little bit personal, but it really. . . I've listened to yesterday and today to people talk about assessments that don't match what they're supposed to do and David talked about sending an assessment to do a curriculums job or vice versa. Each one of your programs has taken incredible care in either crafting or selecting an assessment that answers the questions that were most important to you. If you had to build it, you built it. I think that's a really powerful message to the field about the importance of choosing an assessment that answers the questions that you value. Because a lot of times I think schools and districts have things foisted on them that don't match their values, or people choose tests, somebody was talking this morning, because somebody else chose it. You've put a very, very careful, thoughtful model out there of how you match assessment with the questions that it needs to answer and with the questions that are most important to you, and I think that's a very, very huge gift to the field of how you work carefully.

T.J. Elliott – Thank you. Art.

Art Graesser – I think the major contribution is that this group and effort has demystified comprehension at deeper levels, world knowledge, and the different types of epistemologies that drive that. Before this, I think it was very vague and amorphous and now we're starting to get some handles on, getting us out of the quick sand of flimsy understanding of these deep processes. So I think that's been the major contribution.

Jessica Chambers – I think for me, and I'll speak from my experience with Project READi, is really this collaboration between experts in the field and teachers, especially in the area with that project, specifically, of moving away from the kind of general application of reading strategies and moving toward this disciplinary way of doing reading in middle school and high school classrooms.

T.J. Elliott – I wanted to ask you an additional question before we move on, someone asked you in the hallway when you go back, "what went on at that conference? What's sort of your response?"

Jessica Chambers – [laughing] We definitely spent a lot of time in that hallway talking. You know, well personally for me, a lot of learning about reading comprehension and the different avenues that each project took to get to that. But more importantly, starting these conversations about how we move that research into our schools with our teachers would be a much longer conversation than that, but I think that's what I'm taking away.

T.J. Elliott – That's great. Thank you. Barbara, main contribution of all this work?

Barbara Eason-Watkins - Well, I would really build upon what Jessica shared in that what I heard were a lot of key findings from the various projects that I think could be synthesized in some areas that could really help inform, you know, future work. At the same time there have been clearly some problem areas in practice that need to be addressed moving forward. So that in terms of future research and the need to really extend the learning and the opportunities, we have some direction.

T.J. Elliott – Thank you.

Elizabeth Albro – Ok, this is a hard one. I have a long list. I think, so from my perspective, I feel like we put out a big charge, right? We asked people, we put out a big ask. We said, ok can we construct interdisciplinary teams that look across grade spans, across content areas, that build an assessment that's like no other assessment. And I think that the big contribution is that people did that, right. So when you're a funder you set out these big calls and you just don't have any idea what's going to happen. I think that one of the, sort of for me, one of the real rewards of this is that we've been able to fund this close network of six teams and they have been incredibly honest, like Don said, it's taken a lot of work, but I think we've really learned from each other. We've built bridges across disciplines, across content areas, across grades, and I think that we are in a different place in terms of understanding reading comprehension than we were six years ago, seven years ago, and for me that is a big contribution.

T.J. Elliott – Very good. We're coming close to the end of our time, we've got about five minutes left and I wanted to address two questions here that seemed to be related. One asks for opinions on research on human capacity and development and work place learning. The other one is more pointed. When are we going to stop acting as if materials are the answer, the research we heard speaks to the need for human capacity development. I was wondering if anyone wanted to speak to that particular theme of human capacity development.

Elizabeth Albro – Other than to say we agree.

Jessica Chambers – I think that links to the fact or tends to the fact that this is much longer work. Often we look for quick fixes by putting materials in front of teachers and asking them to work through them exactly as they are and hope for results. Human capacity building takes a lot longer, but it's worth the investment of time.

Don Compton – But I do think good materials facilitate good teaching too. I think it's the marriage of the two that allows teachers to do their craft.

T.J. Elliott – It looks like I've got time for one more question here and I'm going to go with the one that was the latest one, because I think it's something we haven't talked as much about. "Funding efforts should identify programs that target language and literacy for at risk children from pre K through grade 12, not grade range specific." Any thoughts on that notion of targeting at risk children, without making it grade specific?

Eunice Greer – I'll just jump in and say that within the field of assessment, the opportunities that are out there from computer-based assessments to begin to do adaptive testing like Florida is doing really gets at assessing a broader range of students and being able to give you more accurate real information about how kids at either ends of the spectrum, that we weren't assessing well before. We can assess that better now, so I'll throw that out there as one piece.

T.J. Elliott – Wonderful closing comment for us here, as we come to the end. There are questions that didn't get answers, some are specific, for instance there's a question about GISA and I will find a way to get information answering those questions out to participants here, but I want to thank our panel for doing a wonderful job today.

End of Reading for Understanding video.

Video duration: 46:12.