Q&A with a Researcher: Jonathan Weeks

Senior Editor in Research Ayleen Gontz interviewed Senior Measurement Scientist Jonathan Weeks about his work at ETS and his recent appointment as NCME Program Chair.

Jonathan Weeks loves not only connecting with people, but also connecting people. In his 13 years at ETS, he has conducted research for PISA (Programme for International Student Assessment), PIAAC (Program for the International Assessment of Adult Competencies) and K–12; helped develop measures of foundational reading skills; and even worked on some tests for the CIA. His primary focus now is on ECLS–K (Early Childhood Longitudinal Studies–Kindergarten) and other National Center for Education Statistics-led longitudinal studies for middle school and high school students.

You might find him in his office on anchor days, puzzling out a problem on his whiteboard, but you’re more likely to encounter him in the hallways discussing research issues with his colleagues. If you come across Jonathan, stop and say hello. In the absence of research questions, ask him what to stream or read this weekend, talk to him about coaching youth sports or get him to tell you about his mentor.

I saw in your bio on ets.org that you have bachelor’s degree in English from Colorado College. How did you go from being an English major to a job that’s based on mathematical theories?

I always get asked that. I mean, no young kid sits around thinking, “I want to grow up to be a psychometrician.” I thought I wanted to be a high school English teacher, but I didn’t have a teaching certificate. So I ended up getting a job working for a school district with the idea that once I got my certificate, I could go to them and say, “Hey, I've been working for you. Give me a job.” Turns out, I got a position as a secretary in a school district in Colorado Springs, and after several years, I became the primary data analyst in their assessment office.

At this point, I had never taken a statistics course in my life.

I thought, “That's pretty dangerous. I should figure this out.” I went back to school and got a master’s degree in education psychology with an emphasis on quantitative methods. I had ideas about how to address various issues, but nobody wanted to listen to me because I didn’t have a Ph.D. So I went back to school again with an eye toward education policy. I soon discovered policy is really more about politics and decided I was less interested. That was the early 2000s when there was a big emphasis on growth models in K–12 education.

At this point, I thought I wanted to make a better growth model, but ultimately I became much more interested in how scales are constructed. I ended up focusing on issues around the creation of vertical scales, the problem of construct shift, and in particular, the development of multidimensional vertical scales. This is an area of research that still interests me today.

Can you give me an example of the research you do?

I’ve done a fair amount of work on the development of measures of foundational reading skills; the test is called ReadBasix. One of the articles I am most proud of, An Application of Multidimensional Vertical Scaling, essentially provides a validity argument for the multidimensional structure and underlying multidimensional vertical scale for the six ReadBasix subtests.

As I understand it, early in our development, we have a bunch of foundational skills that are all loosely integrated. As we become more proficient readers, those skills twine together to form two cords — word-related skills and comprehension skills — until they eventually form a single cord. The loosely integrated skills correspond to the six factors. These different skills are always there, but they become more integrated to produce the two cords and the more tightly integrated single cord. I was very excited to learn that my research with empirical data provided evidence for a theoretical framework that I previously knew nothing about.

I understand you were one of the program chairs for the recent NCME (National Council for Measurement in Education) Conference. What piece of that planning are you most excited about?

I invited one of my favorite authors to be our keynote speaker. Sam Kean is a science historian and the New York Times bestselling author of The Disappearing Spoon and several other books. In short, he tells stories about science through a series of historical vignettes. These stories are a twist on the history that everyone thinks they know.

Sam is not a “measurement person,” but because of the lens that he looks through, he helps people to think about history and science in a slightly different way. My hope is that as we think about the field of measurement, we can look through a slightly different lens. Can we think differently about the way in which we “do measurement,” the approaches we use to develop tests and the models we use to evaluate the data? Are we going to keep doing this stuff that we've always done, or are we open to trying out some new things that can be more beneficial to the test takers and our field as a whole?

Now that the conference is over, is your job done? 

Technically my job is done now that the conference is done. But I volunteered to stay on a bit longer to try to help out with automating some of the systems. When you are one of the chairs, and your mind’s over here trying to organize this stuff, simultaneously having to worry about extracting and cleaning data… It’s a lot more work than I anticipated. I’d like to see if I can help make this easier for future chairs so they don't have to go through all the time and effort of programming a script or manually sifting through data.

Is there anything in your office you can’t do without? 

I spend a lot of time standing in front of a whiteboard. The way I think about my ideas and the way I process is to stand there and draw things out.

I’m all over the place in the things that I work on, and each of those projects and topics require different methods and strategies. Some are more research-oriented; some are just operational projects that require a practical solution. Even if I am in the office, you might not actually see me in my office because I wander around and talk to people about what they are working on. I call myself a “wandering psychometrician.” Someone might stop me and say, “Hey, we were expecting to have 2,000 kids and we only got 200 for our research. What do we do?” or “I’m trying to simulate data with a seemingly normal factor structure, but I’m getting strange results when I analyze the data. Do you have any idea why?” I enjoy figuring out those more nuanced types of problems.

And sometimes, something that I might be helping out with over in this context actually turns out to be more of a solution for somebody in a different context. I am very much an advocate for interaction and people being able to work outside their specific areas. That said, at the end of the day, I want to help as many people as possible understand as much as possible without interfering with practical constraints — like actually getting our work done.