Raising the Floor. Structuring Public Education

On-screen: [ETS®. Listening. Learning. Leading.]

Speakers: Jonathan Rochkind, Senior Director, Policy Evaluation and Research Center, ETS; Randy Bennett, ETS®, Frederiksen Chair in Assessment Innovation; David Henderson, Retired Professor, Cornell University; Greg Budzban, Dean of Arts and Sciences at Southern Illinois University Edwardsville; Jeff Haberstroh, Principal Assessment Designer, ETS®; Aurora Graf, ETS®; Cheryl Eames, Southern Illinois University, Edwardsville; Peter Van Rijn, Research Scientist, R&D

Speaker: Jonathan Rochkind - As you all know, welcome to our presentation here of Raising the Floor: Structuring Public Education for the Bottom Quartile for 21st-Century Knowledge Work. The theme of this event overall is to build opportunities, structured opportunities to overcome some of the obstacles we are facing, including AV obstacles, which is what we're facing now, but we are not going to let that stop us.

I'm going to introduce the next section here. I'm going to introduce our moderator, Randy Bennett. This next section is about the work that ETS® is doing with The Algebra Project around creating assessments that support the bottom quartile. Randy has been a leader in the field of constructive response and of assessment innovation and has been a real leader on this project as well, and we can't thank him enough. Randy Bennett. (Applause.)

Speaker: Randy Bennett - Thank you, Jon. It's really a pleasure to be here to be part of this conference and to be part of the work that's so ongoing between ETS® and The Algebra Project. It's really been a wonderful experience for those who have been participating in it so far.

We are so excited to have found The Algebra Project because we find that together we complement one another in really nice ways in terms of expertise, in terms of interest, and in terms of commitment to helping kids in the bottom quartile to improve their life chances. It's been, so far, an incredible experience, and we're looking forward to continuing it and for it to continue to be an incredible experience for us.

My role at ETS® is as Frederiksen Chair in Assessment Innovation. The primary thing I do is to lead a research program that focuses on what President Obama has called, not about what I do, but about what he wants: tests that are worth taking and tests that are worth preparing for. That's part of what we're trying to do in collaboration with The Algebra Project, build tests for their curriculum that are worth taking and worth preparing for. This session is going to be in part about that.

The session is going to begin with Dave Henderson and Greg Budzban, who are consultants for The Algebra Project. They're going to focus on demonstrating how The Algebra Project's curricula has developed and the approach they've taken to teaching and learning in the context of that curriculum. They're going to be, I think, showing you and engaging you in tasks and activities that will get you a lot more familiar with that. After their presentation, Jeff Haberstroh from ETS® will talk about how ETS® has approached assessment in the context of The Algebra Project's curriculum. Third, Aurora Graf from ETS® and Cheryl Eames, consultant to The Algebra Project, will talk about how they're jointly approaching designing a universal learning progression for the concept of function in mathematics. Finally, we hope to have some time for questions and answers.

I'm going to ask each presenter as she or he comes up to say a little bit about themselves for those in the audience who have not met them already. Thank you again. Please welcome our first speakers, Dave Henderson and Greg Budzban. (Applause.)

Speaker: Greg Budzban - Thanks, Randy. It's wonderful to be here. I don't want to lead right into doing math problems because everybody is still digesting, so we want to sort of ease into that. I'm going to start by telling a quick story. I'm currently Dean of Arts and Sciences at Southern Illinois University, Edwardsville. I'm going to tell a story about some napkins and some napkins that changed my life. You wouldn't think that napkins necessarily can change one's life, but I have a personal history of napkins who changed my life.

In 2000, I was the Colloquium Coordinator for the math department in Southern Illinois University Carbondale. Bob Moses won the Heinz Award for the Human Condition. We invited him up to Carbondale, and I've had the opportunity and the privilege of hosting him for two days. He was on our campus. As we were leaving town, we stopped for lunch, and I got to listen to him for two days. He actually asked me a question, "What are you thinking about these days?" It was like, "Whoa! Are you really interested in what I'm thinking about?" He goes, "Yeah." I pulled some napkins out and I described something called the road coloring problem. It was a research problem that was unsolved at the time. Drew some diagrams on some napkins, some diagrams that you'll see here in a moment. He asked if he could take them. I said of course. He took them and a couple of weeks later, he called and said he wanted to write an NSF grant because they were about to pull some NSF stuff together. I said of course. We did and it was funded and in some sense, the rest is history.

Fifteen years later, I've been working with them forever since. It's been an absolutely wonderful collaborative experience. We're talking about learning progressions. It's about a method of introducing mathematics through what are called mathematically-rich experiences. It's a pedagogy that Bob has been working on for many years. It has its foundation and things that Piaget and others had done, Quine, and others. It's all about pulling mathematics out of real physical experience. Now, if we had the presentation that I could show you, but what we're going to do is we're going to actually have you do part of this first experience involving the road coloring problem. Now, when we have students actually do this, they actually physically build what we're going to call cities. Now, we don't have time and we don't have the space for you to physically build cities, but you're going to do that at your tables, if that's okay.

I'm going to describe to you some conditions and I want you to try to construct something on that paper that satisfies those conditions. Now that we sort of gently eased into the math, so now I'm going to have you do some math. Here's what's going to happen. We're going to build a city. Together at your tables, you're going to build a city. A city is a collection of buildings. For purposes of this, what's a building going to look like? It's just going to look like a circle with an address.

Here's a building. That's a building, and here's another building, so those are buildings. These buildings are connected by one-way roads. The roads have to be one-way. What's a one-way road look like? It'll look something like that. What you're going to have is I want you to try to build a three-building city that satisfies these conditions. The conditions are the following: Each building must have exactly two one-way roads leaving from it. Each building must have exactly two one-way roads leaving from it. Once you build those roads in your city, connect the city—now, the roads have to go from one building to another. They can't off into space somewhere. They go from one building to another. Once you build your city, you have to be able to follow the roads independent of where you start, so no matter what building you start in, you have to be able to have a path of one-way roads that gets you to any other building in the city. In other words, you can't have any isolated cities. You have to be able to get to any of the buildings in your city. Those are the two conditions.

Three-building city, every building has exactly two one-way roads that leave away from them, and no matter what building you start in, you have to be able to find a sequence of one-way roads that gets to any other building in your city. If you could work on that right now. If there's experienced Algebra Project teachers who feel like they want to just jump up and facilitate because that's what they do, you can feel free to jump up and facilitate.

Part of The Algebra Project curriculum, we do peer reviews, so make sure that you're looking around and sort of working together. We do collaboration. As you build these things, what you have to imagine is actually the circles in a classroom become hula-hoops. They actually become hula-hoops that are on the floor. There's something like a masking tape, a thick masking tape that connects the hula-hoops together. There's arrows drawn on the masking tape. The students are actually physically building these cities in the classroom.

If you've convinced yourself you've done a three-building city that satisfies those conditions, just jump right ahead and do a four-building city, a four-building city that satisfies the same conditions. Two one-way roads that lead away, and no matter what building you start in, you must be able to get to every other building in your city by a sequence of those one-way roads. You got follow the direction of the roads. You can't go backwards. You can't go the wrong way on your one-way road.

[Participants work on exercise.]

We have a question.

Female Speaker - Can we go through a building to get to another building?

Speaker: Greg Budzban - It depends on what you mean, so, yes. To get from one building to another, you can have a sequence (long inaudible section 0:13:14).

Female Speaker - Could you go from building one to building two to building three?

Speaker: Greg Budzban - Yes.

Male Speaker - (Inaudible 0:13:31.)

Speaker: Greg Budzban - That's perfectly fine. It's going to be more complicated for the next step. You've built enough cities.

Female Speaker - Oh, no, we're working (inaudible 0:13:57).

Male Speaker - (Inaudible)

Speaker: Greg Budzban - How many folks feel that they have a well-designed city, that they may have something that they've designed that they can work with? The next step of the problem—question?

Male Speaker - (Inaudible 0:14:48.)

Speaker: Greg Budzban - Perfectly fine. The roads have to go from building to another. In other words, you have to leave the building and get to another building.

Male Speaker - You got to have two roads to each building, and then two roads out, but you have to commit—

Speaker: Greg Budzban - It doesn't matter how many roads go to a building. It only matters how many roads leave a building. You must have exactly two roads leaving each building, because the next step is going to be to name the roads. We're going to have to give directions to move around our city, so we have to give the roads a name. Each building has only two roads leaving from it, so we only need two names. Just for tradition, one will be red and one will be blue. You can choose other names if you wish, but choose two names.

The next step of the problem is to name the roads so that every building has exactly one red and exactly one blue road leaving the building. In other words, you've got two roads leaving the building. One of them must be red and one of them must be blue. It's entirely up to you which one is red and which one is blue. If you have both a three-building and a four-building city, name the roads in both of them that satisfy that condition. Exactly one red and one blue road leaving each building.

[Participants work on exercise.]

You might think about how many different ways there are, how many different ways to name the roads. It's a nice combinatorial problem. Remember, work with your table, do some peer review, make sure that you think you have satisfied the condition.

[Participants work on exercise.]

How many tables feel that they've got a city whose roads are correctly named that satisfy the—

Male Speaker - We are (inaudible 0:18:50).

Speaker: Greg Budzban - Are you? We've got some wizards. Finally, we get to the thing that's actually called the road coloring problem. The road coloring problem was actually originally stated in the late '60s, though it didn't actually appear in print until the '70s. It was stated in an era where there was a great deal of interest in what are called dynamical systems, symbolic dynamics, and it was this. The cities that you've built actually are a mathematical structure called a strongly-connected—and that's the thing where you can get everywhere, that's the strong connectedness—directed graph. A directed graph. The graph means what you have is you have vertices. The buildings are the vertices, and the arrows, the one-way roads, are the directedness of the graph, the edges that are directed. Those buildings are strongly-connected directed graphs.

The question was, because of some issues in dynamical systems that, as we say, are beyond the scope of this talk right now, but because of some issues, they're trying to figure out the dynamics of it, the question was, can you name the roads so that there is a synchronizing instruction? What's a synchronizing instruction? Synchronizing instruction is a sequence of labels of road names that no matter what building you start in, the same sequence of instructions will get everyone to the same building at the same time.

Imagine there's a party. There's party at building three. Everybody needs to get to building three. You don't want to give everybody different directions. You want to give everybody the same directions. Say, "Take red, red, blue." That means no matter what building you start in, you're going to find the red road from the building and follow it to the next building, and then from that building, find the red road, and then from that building, find the blue road. What you want is a set of instructions that gets everyone to the same building at the same time, so you synchronize everyone to one building. Now, what I want you to do is see if the cities that you've designed with the road addresses, the road names that you've assigned, do they have a synchronizing instruction? What does that mean?

Male Speaker - (Inaudible 0:21:51.)

Speaker: Greg Budzban - Very good. Thank you very much as always, Bob. Bob knows best. What we have is the students actually physically move these things. We don't have students to physically move it, you haven't drawn it, but what you have is, you have candy. Each one of those pieces of candy, you can put it in one of your buildings and they represent a person. You can move them around to see if you can get them all together in one place. Let's spend a little bit of time seeing if you can find a synchronizing instruction. If you cannot, and I've looked over your shoulders, and there's some of you who will not be able to, see if you can figure out why. If the road coloring that you currently have does not have a synchronizing instruction, in your opinion, is there a way to change the names on the roads so that it does, or maybe you've got a structure where it's impossible.

[Participants work on exercise.]

I hear some questions in the background. Everyone has to move together. When you say "red" everybody in the city has to move at the same time. In other words, everybody is moving at the same time when you say "red." When you say "blue," everybody in the city is moving to finding the blue road in the building they're in and moving. It's all at once. All the movement is at once.

Male Speaker - (Inaudible 0:24:25.)

Speaker: Greg Budzban - No, it's not impossible.

Female Speaker - (Inaudible 0:24:38.)

Speaker: Greg Budzban - If you have another building, but not with that structure.

Male Speaker - These move, too.

Speaker: Greg Budzban - That's impossible, but why? It does not matter what city you end up in. Wherever you happen to land, it's fine.

[Participants work on exercise.]

We have to move on. We can't do this. Now, usually, what we're doing right here takes about three days in a classroom to go over, so we're condensing all this.

[Participants work on exercise.]

Does anybody have a solution? We have a solution.

[Participants work on exercise.]

Again, we've got to move on. If you want to continue thinking about this, I'll be around.

Just some history. As I said, this problem was posed in the late '60s. They were certain it was going to be solved within two years at the most. The answer is, does every strongly-connected directed graph that has another property that I didn't describe, does it always have a synchronizing instruction? They thought they were going to solve it quickly. Forty years later, it was not solved. A few years ago, it was recently solved by Traubin (sp? 0:28:55). The answer is yes, so that you can always find a synchronizing instruction if you have a strongly connected, aperiodic, that's the one that I didn't tell you about, aperiodic directed graph.

Some of the structures you drew were not aperiodic. They were periodic, and that's why some of them don't work, but you can always find a coloring of the roads that works. In other words, if there's a billion buildings, a trillion buildings, a billion/trillion buildings, and it satisfies those two conditions, there's always a way to name the roads to find a synchronizing instruction, which is amazing thing. Does anybody want to come up and show their example? Do you want to draw it up here, Ben?

Male Speaker - Yeah.

Female Speaker - We will admit, we were at a little advantage because we were at Bob's table, and Ben believes there's osmosis in learning this mathematics thing because he's been with Bob so long, but we got there with a lot of help. This is harder without the candy on the table. That's what Bob helped us have is we took the candies.

Male Speaker - I'm trying to remember how we did this. I think we started with red. Move red road first, and then move red road again, then—we might need some help here, folks.

Female Speaker - We were in blues.

Male Speaker - We started blues and blue again. That's just moving us—and then red. That's how we did it.

Male Speaker - There's a problem with your city.

Male Speaker - There is. What's the problem?

Male Speaker - You can't get to building four.

Male Speaker - This is how you cheat. (Laughter.) What can I say?

Speaker: Greg Budzban - (Inaudible 0:33:48) in the city, that gives you an idea of what the synchronizing instruction is. That physical activity, students actually moving around in the classroom with cities that are physically constructed, that physical movement is actually the first step of a learning progression called APOS Theory. APOS stands for Action, Process, Object, Schema. I'm pulling this right from Ed Dubinsky's website. Ed Dubinsky is the author of APOS Theory. It says, "An action is a repeatable physical or mental manipulation that transforms objects." That's the first part of a learning progression that was in the back of my mind as I wrote the Road Coloring module where—and I think is really at the heart at some of the stuff that we do in The Algebra Project, starting with a physical experience that is concrete to the students where some kind of transformation occurs. What you experienced somewhat, not the way the students do in the classroom, is that first part of a learning progression that's at the foundation of some of The Algebra Project curriculum. Thanks very much. (Applause.)

Speaker: David Henderson - I'm going to lead you on an example from geometry. I'm David Henderson. I was a professor at Cornell. Now I'm retired from Cornell. I'm the geometer on The Algebra Project. Again, start with some physical experiences. One of the basis is symmetry. There are two activities I want you all to try at the table. Get a piece of paper. The graph paper that you have there will work actually better than the lined paper. Draw a vertical line. It'd be best if you draw along one of the lines on the graph paper just to make it easier.

What we want to talk about is symmetry. I had slides with some examples. Symmetry, so examples that we're familiar with is the letter A has reflection symmetry around this line. One of the ways you can check that is to trace this on tracing paper. We use Patty Paper. Those of you who know what Patty Paper is. You just flip it over and see if it coincides or our hands are roughly mirror symmetry.

There's another kind of symmetry also, which is not so common, and that's the symmetry of the letter S. The letter clearly has some symmetry in it, but it's the symmetry that often doesn't get talked about in school. It's half-turn symmetry. If you rotate it half a turn, it comes back itself. If you draw on some transparent paper and rotate that, it comes back on itself. Half-turn symmetry turns out to be very powerful in geometry. All the stuff that we torture students with about parallelograms, they're all summarized by saying that parallelogram has half-turn symmetry. To get an experience of what symmetry actually is, so embody it and see it more as a function, now I go back to this drawing this line down. Now, get partners. Two people work together. One person start on one side of the line and slowly draw a curve. The person on the other side follows that person and draws the curve on this side so that the result has mirror symmetry.

Speaker: Greg Budzban - Dave and I are going to demonstrate it real quick here for you.

Speaker: David Henderson - The first person has to go slowly.

[Participants work on exercise.]

Once you've done that, this is something that often people have trouble with the first time, but it's something that with a little practice, you can get better and better. You're really feeling what symmetry is and as a function and as a transformation. Let's also while we have your attention, then we'll go around, do the same thing for half-turn. For half-turns, there's essential point. Again, one person is going to start drawing a curve. This is harder than the mirror symmetry; at least I think so.

[Participants work on exercise.]

Not such a good job. Again, you can check it by rotating it around. The more experience you have, the better. Now try it, both of those. You got to get in pairs and take turns who's leading and who's following.

[Participants work on exercise.]

We need to move on. We're out of time. Particularly, if you actually use the graph paper so that you're going along the lines of the graph paper so that you're doing some broken curves, there's some structure to it. It's not something that you can actually—there's nothing to memorize there. There's no facts to learn. Well, maybe a few you could talk about, but it's something you need practice to be able to do.

This is one of the things that we're presenting to the ETS® as challenges. How do you write machine-gradable exams that'll grasp this kind of stuff? They said they can do it.


Speaker: Jeff Haberstroh - Hi, everyone. I'm Jeff Haberstroh. I work here at ETS® in the assessment division. I'm a principal assessment designer. Prior to coming here—I've been here close to 30 years now—I taught math and science high school and middle school levels. One interesting little thing. I just got invited to help out on this project probably about six or seven months ago this past spring, and that's what I want to talk about. I'm going to talk very briefly about some of the assessment development activity in the next couple of minutes.

One interesting little thing, when I first started here, a year or so after I started here, I also was working on an algebra project at that time. It was a joint activity between the College Board and ETS® under the—it was called The Educational Equality Project, back in the late 1980s. One of the objectives of that project was to improve access for students, because first-year algebra was seen as a gatekeeper course, even way back then. A lot of the thinking that's happening here really does go back for several decades. A little other tidbit, one of your colleagues in the audience, I had the pleasure of working with Nell Cobb way back then, so we've come full circle on this project, here I am on algebra again.

I just want to take a couple of minutes right now and talk to you a little bit about the development of an assessment for the project. What happened was probably late last winter or early last spring, Randy came to us. There'd been some conversations with Dr. Moses and some others on The Algebra Project, and there was some real interest in getting some assessments in place so that students who completed the modules like Road Coloring, and another model, Racing Against Time, would have an opportunity to really demonstrate what they knew, the mathematics that they knew, in a more standardized assessment environment, if you will. That is, come in, sit down for a timed administration, and answer some questions. We were presented with a somewhat challenging task that David just alluded to for Racing Against Time and one of the other modules, Road Coloring, to figure out exactly how we could really capture the essence of what those two modules were about in terms of mathematical content and give the students a fair opportunity to demonstrate what they know.

I don't know how many of you know a lot about testing or the process of testing or assessment development, but very briefly, what we had to do first is we had to really think about what the focus of these assessments were going to be in very specific terms. I should mention that this whole process since last spring has been a genuine collaboration between people on The Algebra Project and folks here at ETS®. I'm working with a team of four or five other people in the assessment division here to work on the developments of the assessments for these two modules.

We actually came up with what I think are a very effective working set of specifications or guidelines on what to include in the assessments in terms of numbers of questions, what the emphasis should be, and so forth. One of the particular parts of the specifications that I found very helpful, as have my colleagues in development, is this idea of close transfer versus farther transfer of mathematical knowledge that the students gain by working for the assessment. As Greg mentioned earlier, one of the driving ideas behind the modules is that we want to try to give students the opportunity to show what they know about the mathematical content, but doing so in a context that they're familiar with, so we carried that idea over to the assessments themselves in terms of the specifications.

We're developing questions in Road Coloring, for example, that are—60% to 70% of the questions in the assessment are tied very closely to the Road Coloring context. Within those 60% to 70%, there are varying levels to where for some questions, students show how familiar they are with the language of mathematics, because mathematics is a language just like English and Spanish and any other language. One of the first things all students have to be able to do in order to be successful in mathematics is they have to know the language. What we do in the Road Coloring module is we assess their ability to work with representations like the arrow diagrams that you all just worked with that Greg showed you, and other representations like ordered pairs, directed graphs, coordinate graphs, and things like that, just different ways to represent functions.

There are other general areas of mathematics that are also tied to that Road Coloring context in there, but we don't stop there, because even though we're focusing on the first quartile of students here, any good assessment always has a goal of trying to differentiate among the target population of students. What we're trying to do here is with the focus on the first quartile, we also want to see how many of those students can work beyond the context of the Road Coloring setting, so we do present them with some additional context to work with as part of the assessment.

The other assessment, Racing Against Time, we do something a little bit different in there. Randy mentioned at the beginning about CBAL, which is one of our major research and development projects here at ETS®. For those assessments, they're all delivered online. We're incorporating one of the online extended tasks from CBAL into the Racing Against Time assessment. We did that with the Good Housekeeping seal of approval of The Algebra Project folks who are collaborating with us because all of the content from that particular CBAL online task really aligns well with the content of the Racing Against Time curriculum module. We're looking forward to seeing how that works out.

I should mention that CBAL shares, I think, a lot of the same basic principles and philosophies on how students should be assessed because—I work on CBAL also, by the way. That's been one of my major assignments here for about the past seven years, in CBAL mathematics. The whole notion on CBAL about presenting and assessing students' mathematical understanding in contexts that are meaningful and relevant to them, those principles are not unlike the principles exemplified by The Algebra Project in the Road Coloring module and its other module, so there are a lot of commonalities there.

Where we're at right now, very briefly, and I'll close. As I said, we started about April or May of last year with the specifications. We then took the development of the assessments through the standard ETS® assessment development process, which means we have people work on writing questions, and there are extensive rounds of reviews. We review all the questions internally to make sure that the content is accurate, it's assessing each of the questions, or is assessing what we believe they should be assessing. The Algebra Project folks also do content reviews in tandem. In addition to that, we have other levels of review; one for fairness and sensitivity to make sure that our questions are not biased. We have reviews to look at the editorial style and things like that.

Prior to finalizing the assessments, we do some informal tryouts. We just completed a couple of rounds of informal tryouts for the Road Coloring assessments. We had a couple of hundred students take those assessments. We reviewed the results. We made some revisions to the questions, which I think made it even better. It's always great to get student input via the tryouts when we do that. Right now, we're just about finished with the Road Coloring assessment, ready to put that one to bed, to have it printed. We're about halfway through the development process on the Racing Against Time assessment. We're headed for tryouts with that one early February of next year. That'll be ready for the students early spring or so. That's about it. Thank you very much.


Male Speaker - Jeff. CBAL, is that kind of like a beach ball?

Speaker: Jeff Haberstroh - C-B-A-L, that's an acronym. I work on it so much, I expect everybody else knows automatically. Cognitively Based Assessment of, for, and as Learning. As I indicated earlier, it's a long-term ETS® R&D project. Basically, grounded in learning progressions research. Everything we do, we develop extended tasks, supporting materials for teachers, do professional development workshops with teachers, all around the idea of learning progression. I think one was mentioned earlier by Greg or Randy. An example of a learning progression is linear functions and the idea of slope, and trying to understand how students develop, and understanding of slope as a rate of change. Why are we doing this? We're doing this basically to provide students, teachers, and others with useful information about how students learn key mathematical concepts and ideas. Any other questions right now?

[Participants give inaudible responses.]

Speaker: Aurora Graf - You're not going to be able to see my screen, but we're just going to use them kind of like index cards. We're going to talk about the DR K-12—Discovery Research Pre-K-12 proposal that we've been working on. This is a collaboration between The Algebra Project, the Young People's Project, Southern Illinois University at Edwardsville, and ETS®.

The project goals include formulating a universal learning progression for the concept of function that addresses crosscutting mathematical themes and practices. The idea behind formulating a universal learning progression has to do with the fact that the Road Coloring materials are based on the concept of function, but they use some novel representations like the strongly-connected directed graphs that aren't in most standard curricula. There are other more standard curricula, and they have a different instructional sequence and use different kinds of representations. We need a learning progression that will work for students regardless of the instructional sequence, so that's the idea behind building a universal learning progression.

To this point, at ETS®, we've developed learning progressions for crosscutting themes, like mathematical argumentation and mathematical modeling. Gabrielle Cayton-Hodges and Leslie Nabors Olah have been working on those. We have quite a few content-based progressions, but we haven't really tried to integrate the two. That's the rationale for also addressing crosscutting mathematical themes and practices. We'd also like to develop innovative and interactive computer-delivered tasks intended to assess student understanding with respect to the learning progression. In our experiences on the CBAL project or research initiative, we've developed quite a few computer-delivered tasks, some with simulations.

We'll be conducting cognitive interviews with students as they try out the tasks, and then we're going to apply existing automated scoring technologies, and we'll also develop a new scoring engine to assign levels of the learning progression to task responses. In other words, we're going to score the tasks using the levels of the learning progression. For many of the tasks that are based on The Algebra Project materials, we are going to use automatic scoring technology to do it.

We will administer a small-scale pilot after the cognitive interviews, followed by a larger scale field test, and then we will evaluate the empirical recovery of the levels of the learning progression based on the results of the field test. Finally, in all the steps above, we're going to consider how to engage students who are segregated in schools by circumstances that are linked to race, class, and ethnicity, and are characterized as underserved in terms of their performance in mathematics.

The CBAL working definition of learning progressions is as follows. In CBAL, a learning progression is defined as a description of qualitative change in the student's level of sophistication for a key concept process, strategy practice, or a habit of mind. Change in student standing on such a progression may be due to a variety of factors, including maturation and instruction. Each progression is presumed to be modal, that is, to hold for most, but not all students. Finally, it is provisional, subject to empirical verification, and theoretical challenge. That's from Deane, Sabatini, and O'Reilly, 2012. Cheryl is going to talk a little bit about background work on learning progressions.

Speaker: Cheryl Eames - Hi. I'm Cheryl Eames. I'm from SIUE with Greg. I've spent a few years working on some learning progressions with the research team at University of Denver, Doug Clements and Julie Sarama, and a team at Illinois State, and my dissertation work involved learning progressions.

When we build learning progressions, we do an extensive review of the literature, we conduct teaching experiments and learning progression-based assessments, but what I really wanted to talk about is why we care so much about a learning progression when we're building an assessment, or developing curricula, or when we're doing professional development. The main goal here or the main thing is that we're looking for a progression or a spectrum of thinking instead of just thinking about an answer being right or an answer being wrong. That's a major shift. That's a huge advantage instructionally. We know that it's really helpful for kids when we can do that. We know that when we develop curricula that are based on learning progressions, it has a positive impact on student achievement.

Our Common Core State Standards for mathematics were informed by what we knew about learning progressions at the time. We know that learning progressions are really effective tool for professional development. When teachers can see levels of thinking in their classroom, it really helps them capitalize on that thinking that they see in their students and they can have productive mathematical discussions. They're also a really important tool for follow-up research. Once we have a learning progression that we understand that we know is valid, then we can go in and start thinking about how children at different levels react to instructional interventions, so it's a really important tool for researchers, as well as teachers and has far-reaching potential impact beyond just assessment development.

Speaker: Aurora Graf - I'm going to talk a little bit about the concept of function learning progression, which we've developed here at ETS®. We're going to use it as a starting point in the DR K-12 work if it's funded, but we also recognize that it's not really universal because it assumes a more traditional instructional sequence. Although learning progressions are primarily theoretically-based, they are also—it's not as if they're not affected by the curriculum or by the sequence of instruction. That would be our starting point is the ETS® learning progression.

I'll just talk about the levels of it, as it exists right now. The first level one is pre-exposure. There's no concept of function yet. It hasn't developed, but students can extend sequences, for example. At level two, they become familiarized with functions, but they think of them as indistinguishable from formulas. A function is perceived as a computational process or an algebraic expression. If you have two equivalent algebraic expressions, the student will see them as two different functions. At level three, students start to make connections. They see a function as a rule, and the concept of a dependence of one variable being dependent on another is beginning to develop, but the notion of what Vinner and Dreyfus refer to as one-valuedness is not yet firmly in place.

At level four, which is the synthesis level, the one-valuedness idea has developed and the students are facile in the use of alternate representations of function, so they can translate between equations and graphs and tables. In the case of The Algebra Project curriculum, they're also translating between strongly-connected directed graphs or isometries. Students can consider how variables co-vary and attend to global features of graphs. At level five, students perceive a function as an object that can be operated on in its own right. Finally, at level six, they're drawing extensions by thinking about functions as families; so, families of functions are perceived as parameterized objects, and the role of the parameters is understood and the role of domain and range is well recognized.

The next step after that would be task development. The goal here is to develop innovative and interactive computer-delivered tasks that are intended to assess student understanding with respect to the learning progression. The innovative tasks that we develop are going to be based on The Algebra Project materials. Conducting cognitive interviews. We'll also be conducting cognitive interviews with students as they try out the tasks. The development phase and the cognitive interview phase is going to be iterative, so we expect it will take about 18 months for these two activities combined. It's going to be an iterative process where we develop some tasks, we conduct some cognitive interviews, we realize ways that we want to change the tasks or new tasks that we need to develop, and we do some more.

Here's an example of how we would use automatic scoring to score some of these new innovative tasks. This is an example from Greg's Road Coloring. The task for the student here is to take a city representation and convert it to an arrow diagram. I guess you can't see my screen. I always forget that people can't see my screen. (Laughter.) Anyway, you can represent what's in a strongly-connected directed graph as a pair of arrow diagrams, one of the pair for the blue roads, and the other one for the red roads. In this kind of activity, it would be interactive because we could, for example, give the student the graph and have them create the arrow diagrams; or we could go the reverse, give them the arrow diagrams, have them create the graph; or we could even have them do both. Then we can automatically score these to see if, say, the arrow diagrams are consistent with the representation given in the directed graph.

The way that we would go about doing this—and here I want to acknowledge Jim Fife, who came up with the idea, he's the automated scoring expert for mathematics—that all of these representations can be encoded as matrices. We could encode the directed graph as a matrix, and that would be the key. The arrow diagram that's produced by the student could also be encoded as a matrix, and then the matrix that's the key could be compared to the matrix that corresponds to the student-produced directed graph. If they match, it's correct, and if they don't match, then it's incorrect, so then all we have to do is figure out how to assign different kinds of mismatches to levels of learning progression.

This is our proposed cycle for validating a learning progression. This appears in our chapter. Again, you can't see it. Stephanie has a poster that has this graphic on it that's going to get handed around, but just to summarize it, there are several steps involved in validating a learning progression.

The first step is really more of a theory development step where you analyze the research and the logical structure of the domain so that you have a strong foundation for the design of the learning progression. The next step is to evaluate the empirical recovery of the level. In other words, the learning progression is a theory about how student understanding is sequenced, so you can ask questions like, are the descriptions that we've given of these levels of understanding ordered correctly, are they all necessary, are they sufficient? These are the kinds of questions that we're interested in answering in a study of empirical recovery.

The next step is to compare two competing models. Even if you empirically recover the levels of the learning progression and the results are consistent with the ordering that you hypothesized, there still could be other theories of learning that account equally well for that support, so then you need to consider what might some of those other theories be.

The ultimate test for learning progression is whether when you use it in the classroom, is it instructionally effective? That would be an evaluation of instructional efficacy. We have this as a cycle, because you might go around several times, but in the meantime, you can use the learning progression provisionally and the design of forms and tasks. Peter is going to talk a little bit about the empirical recovery.

Speaker: Peter Van Rijn - Let me first introduce myself. I'm Peter Van Rijn. I'm the psychometric lead in the CBAL project. I work with Aurora on learning progressions. We have worked on several of them both in mathematics, linear functions, and proportional reasoning, but also in English/language arts in argumentation. We have several studies in which we try to find empirical support for these learning progressions, and in this project for the learning progression about the concept of function.

The main thing you want to find is that this ordering is consistent across the different items that we have developed. So, the mapping of the item responses to the levels of the learning progression, you want that to be consistent across the different items. If you map for one item a response to a level two, you want to find that same mapping to a level two response to be at the same level. There are several statistical methods that you can use to check those empirical orderings across the different items. Sometimes, you find items where these levels are reversed, so you might want to revise your mapping to the different levels. If overall there are some issues with certain levels, so as Aurora was mentioning, if the levels are not distinct enough to make any statements or make any inferences with respect to that level, you might need to revise your theory a little bit or it's not what you think is the case with the target population. That's some of the problems that we are working on and will be working on in this project to come up with empirical support for the theoretical learning progression. I think I'll leave it at that for now. You want to finish up or…?

Female Speaker - We were just going to open it for discussion.

Male Speaker - Thank you very much for the presentation. I'd just like to begin the questions with some basic questions about what you said. Aurora, could you give just a brief description of what you mean by automated scoring? When you said that these will be scored using automated scoring, are these paper-pencil administered assessments, or will they be a machine computerized assessment? What do you mean?

Speaker: Aurora Graf - Ultimately, they'll be delivered by computer. When we do the cognitive labs early on, they'll be mostly paper and pencil with a small number of computer-delivered ones that we'll try out, but the goal is to get them all computer delivered by the pilot test. Most of them will be relatively short-type questions. It's The Algebra Project materials that we're going to develop the more interactive tasks for. The interface will allow students to, for example, draw arrow diagrams or the graphs or to draw figures and even enter matrices.

Male Speaker - People won't score these, right? What do you mean?

Speaker: Aurora Graf - Some of the items, for many of the items, we will need people. For those that are constructed response, where they're entering a short text response there, we will definitely have people scoring them. We'll also be looking at using HENRY, which is ETS'® scoring engine for short text responses so that we'll be able to compare agreement between one human with another and a human with HENRY. For example, for the item that we just showed from Greg's Road Coloring, that can probably be scored with extremely high accuracy by machine.

Male Speaker - Just one other question. When a student receives the score, what would it look like? What kind of information will you be giving the student?

Speaker: Aurora Graf - That we still have to think a bit more about. We haven't included as an explicit part of this proposal to think about score reporting, but it is something that Peter and I have started to think about for learning progressions in general, which is what sort of representations should you use to convey certainty that a student is at one level versus another. That would likely be accompanied by some sort of text description of what it means to be at that level. It's possible we can consider some provisional text about what you might want to try next. Although, we'd have to give some careful thought as to at what point we would want to introduce that text. Do we want to do an evaluation of instructional efficacy before we do that? That's something we'd have to think more about.

Male Speaker - I have two questions. One is probably the easier of the two. The rollout of the Common Core assessments, as I'm sure you know, has not been very smooth. One of the major issues has been the reliance on computers for delivering the assessments and the claim that in a lot of states they simply don't have the technology. Are you concerned about that issue? If so, what do you think?

Speaker: Aurora Graf - We have included in an early phase of the project to administer existing computer-delivered tasks just to see how familiar participants are with computer delivery in general. We'll have some time to notice those aspects that may make tasks construct-irrelevant. Also early on, we're going to deliver a few of the tasks that we're developing for this project, not all of them because we're going to need more time to get those all developed, but those few that are developed, we're going to administer towards the end of the cognitive labs.

That will give us also more information about any construct-irrelevant difficulty that's been introduced by virtue of computer delivery, and will give us a chance to make revisions. It's not a guarantee that by making those revisions that there still won't be difficulties that occur with using the technology, but we can—the goal is to get it to a point where it's both more efficient and more enriching than using just paper and pencil.

Male Speaker - Can I ask one more question? You talk about learning progressions, but what I heard you say about them, what I would think of as stages, because you didn't talk about how you get from one to the next. Is there a way in which assessments could help students do that?

Speaker: Aurora Graf - The goal of CBAL is cognitively-based assessment for, and as learning. The "as" part was an important part of the CBAL work, where you hope that students learn as they are doing the tasks, so that doing the task is a learning experience in and of itself. We'll need to give some careful thought as we design the tasks. Are we designing this task in such a way that it will move students' thinking from level one to level two, or will move students' thinking from level two to level three as they engage with the task? Hopefully that's something that will come out in part from the cognitive interviews.

Female Speaker - I have a question about the use of technology. I'm from the city of Boston, and I've been in and out of dozens and dozens of schools there. The schools just don't have the capability to have students take tests using the technology, and also the students by and large haven't had the level of experience, so they've developed the facility to take tests using the technology. How do you deal with these two sets of limitations? Does that make any sense?

Speaker: Aurora Graf - As part of doing this project, since we're administering the cognitive labs on paper and pencil, we will still have the paper-and-pencil versions of the tasks available and we can still use those. For example, if we have a task that's based on a social media context, for example, and the students are not familiar with social media, then—

Speaker: Female - (Inaudible 1:23:22) social media using their smart phones. That's basically (inaudible 1:23:32) access Internet.

Speaker: Aurora Graf - If there are other aspects that might be unfamiliar, like the graphing technology, for example. There are other ways to do that. There are ways to do that on paper and pencil.

Speaker: Female - Thank you for the paper. I'm just curious to know what do you hope happens? What do you hope the outcome is? If your goal of the study plays out, what do you hope to do with what you find from this study? What audiences do you think would be or are interested in the outcomes of the study? I think this is really interesting, so I just wonder what you hope happens and what audiences you think could utilize what you find from the study?

Speaker from audience: Stephanie Peters - Hi. I'm Stephanie Peters. If anybody else wants one of these, right here, it's a paper. It's just a little bit more about the concept of function learning progression and the initial cog labs that Ed (inaudible 1:25:12) the initial step to develop a task for the concept of function. In doing this, we just looked at, hey, when students answer questions on this task, is it what we think the learning progression based on the theory and based on the standards is meant to look like? Do we need to look at the task and think about a different way to refine it in order to get different evidence of what they're thinking, students are thinking about? Or do we need to look at the levels of the learning progression in the bottom left corner of this very tiny—apologies—is what was discussed with regard to (inaudible 1:25:56) cycle and the empirical recovery? It's a very early step and it's difficult to read, but (inaudible 1:26:03). Aurora, I just wanted to say that and thank you for passing that.

Speaker: Aurora Graf - What would be sort of the hoped-for outcome and who would be interested in that outcome? The best outcome would be if the students can meet a level of understanding that's at least commensurate with the standards. At the very least that they have the skills and the information that they need to proceed in the best way possible. Who is this for? Of course, it's for the children, trying to prepare them for further learning and achievement.

Speaker: Greg Budzban - If you don't mind, I will—

Female Speaker - (Inaudible 1:27:42.) As a parent, can I take this back and let other parents know that (long inaudible section 1:27:53) ETS® is working on evaluating (inaudible 1:28:04). How can I takes what's here, and once your study is complete have some action around it? That's what I mean by what (inaudible 1:28:14).

Speaker: Greg Budzban - Let me address that. Certainly, the idea of building curricula, mathematics curricula, that actually manifests the way students progress in their learning, so trying to really understand what the levels are in terms of this slowly more sophisticated understanding of one of the most fundamental and important ideas in mathematics, which is a function.

If they're going to be successful at the university level in mathematics, they've got to understand what a function is. What we hope to get from this is really to understand how students progress through the ideas that get them the sophisticated notion of a function that is going to help them be successful, for example, in their later academic career. That's the first thing. The second thing is who's it for? The DR K-12 grant that we're about to put in is arguably unique in a sense that the students that we're going to be working with, in terms of the cognitive interviews and the pilot tests and the field tests, are really students at the bottom quartile. We're not working with a select group of students. We're really going to be working with students in the group of students we're most interested in advancing through the academics, so getting their feedback on how they're learning is going to be really essential. Once we develop this universal learning progression, which I'm sure we will be successful at, we really be able to transform the way that idea is taught, but for all students.

Male Speaker - (Inaudible 1:30:19) the name of this, CBAL. By trade, I'm an old coach. When I think about CBAL, I think about taking all of our athletic energy concepts and try to shift those into academic synergy working with our kids at the bottom of the cortex, who have no real passion for learning. You take a concept like this after it's been developed—this is going to sound kind of crazy, but you're talking about a universal concept. Why wouldn't we approach Nike or Adidas to develop a CBAL shoe and cap and jersey and have a national competition for the kids who are not performing? Let's find a way to get them energized to learn this concept and put it to use in the way it would be relevant to them. We crystallize athletics. We need to start trying to crystallize academics. All of that athletic energy that our parents put into their kids early, we need to help them to begin to understand to move that towards academic synergy.

This kind of concept, with the levels of kids increasing their skills, would be a great competition. You get a Nike, and it would be called a CBAL competition, where we go from school to—and I'm from the state of Texas. We're big on athletics. Now, you'd have to get by TEA to do something like this, but why wouldn't we go to Nike or Adidas or somebody and say, "Hey, we want you to sponsor a national academic competition"? Let's start crystallizing—because those same kids buy Nike shoes even though they don't play any ball and even though they don't shoot not one jump or throw one pass, they still buy Air Jordans and LeBron James. I'm an early advocate. I tell parents, "Let's quit buying LeBron James and Air Jordans and start buying some books," but that's another story.

Why wouldn't we take a designed solution—this is a solution. Why wouldn't we take this solution and market it so that it becomes common practice across the United States, and we have a state-by-state competition with kids in that lower quartile, that if they increase their skill sets—like we play Tuesday night basketball and Friday night football. Let's have academic competitions where kids begin to compete academically and there'd be awards for them. It's just a thought.


In Texas, we have the STAAR test, which is basically content pulled from the TEKS, and design a logo with the STAAR test, and kids with football cleats on chasing math and science heading towards the goal line. These STAAR test books with this mathematics, got big drops of sweat coming off of them because those kids about to capture, take it over.

Let's do something creative. When you sit here and come up with these great ideas, we sit on them the wrong way. You got to blow this stuff up. You got to blow these ideas up. If you come up with a great idea, and instead of us—when we sit in rooms like this, we ought to be creative. Bring people with creativity in the room to suggest how do we take a great idea that you've worked hard on for years or months, however long you work on it, and it works, but we don't get it out to anybody, so very few people get to use it in practice. When we could blow it up and get some of these people who our kids are spending all this money with and say, hey, we want you to do this competition and have a CBAL national championship. Do a state-by-state competition where the kids in the lower levels are competing to increase their score because they're going to get a pair of Nikes or a warm-up or get Nikes—and then having them to put some money behind it. The school, the state champion in the state of Texas, if you win, you get $50,000. Here in New York, the winner gets $50,000. We bring all of the state winners into a big national championship and there'd be some money for the school district that wins or the state that wins. Ideas like that is what we got to move towards, and stop sitting on these great concepts. (Applause.)

Speaker: Greg Budzban - Let me just follow up on that real quick because I had a conversation yesterday. I don't know if anybody in the room recognizes the name Mannie Jackson, but Mannie Jackson is the owner of the Harlem Globetrotters. He's a native of Edwardsville. I had a conversation with him yesterday and he wants to build a national collection of centers around academics. Of course, Bob can tell you all kinds of stories, Myesha (sp? 1:36:47) can tell you all kinds of stories about math games that have already been developed by The Algebra Project that are exactly what you're talking about: getting students involved with physical activity that actually embodies mathematically-rich ideas. We just need the right spark, and so I'm just right behind that idea. I just wanted to let you know.

Speaker: Dave Henderson - I agree with that. I wanted to say why it'll be interesting to me if this project is successful is that the various curriculum ideas in The Algebra Project are getting at things very differently than in the standard curriculum. If we can understand what the relationship is between what The Algebra Project is doing and what the standard curriculum is, it'll help get things—right now, it's very different, so it's very different for school districts to understand we're doing something that we're convinced will give (inaudible 1:38:00) experience, and it gives a solid foundation for the students so that they're going to end up in a better place than they would otherwise. They're doing something that's not part of the standard curriculum. Therefore, it's not covered by the standard tests, etc., and that's a real problem. If we can understand that relationship and present it in a way, it'll help discussions with school districts and just help in thinking about these things as part of the curriculum.

Male Speaker - I think it's about time for us to start wrapping up. Did you want Randy to make some closing remarks? Maybe you can go to (inaudible 1:38:39).

Speaker: Randy Bennett - My only closing remarks are to thank you for your time and attention and great questions and great discussion and to ask you to thank our panelists for their presentations.

Male Speaker - There's more, everyone. Thank you so much for joining us tonight. Tomorrow at eight a.m. we'll be serving breakfast. For those of you who are up even earlier than that, we have breakfast in the Solomon Dining Room across, and then we'll have a wonderful day talking about Raising the Floor and improving educational outcomes for the bottom quartile. Very excited about that. Some of the panels I'm particularly excited about is the one that we're having with the teachers. Those of you who are teachers here for The Algebra Project, can you stand up for a moment? Great.


On-screen: [ETS®. Listening. Learning. Leading.]

End of Raising the Floor - Structuring Public Education video.

Video duration:  1:39:57