SuccessNavigator™ A User's Guide to the Assessment

People in this video

Narrators/commentators as noted below in text.

Narrator – Ross Markle, Senior Research and Assessment Advisor for the Educational Testing Service

 

Transcript Body

On-screen:[ETS® SuccessNavigator™ Improves Retention and Completion Rates]

On-screen:[ETS® SuccessNavigator™ A User's Guide to the Assessment]

Narrator – Ross Markle: Hello, I’m Ross Markle from Educational Testing Service, and I’m here today to talk to you about a new offering we’ve developed called the SuccessNavigator™ assessment. This represents an exciting new endeavor for ETS®, as we seek to fulfill our mission of advancing quality and equity in education for people worldwide, in this case by working to improve the retention and graduation rates for students in higher education, in both two- and four-year institutions.

On-screen: [AGENDA:

I first want to talk with you about the current climate of student success in which far too many students leave higher education without receiving a degree or certificate. I’ll also talk to you about holistic assessment, which seeks to understand both the academic and noncognitive factors that influence student success, and how this can be a powerful tool for helping more students persist to a degree.

I’ll then introduce you to the SuccessNavigator™ assessment — a test of noncognitive skills that can be used by students, advisors, faculty, staff and institutions of higher education to better understand student characteristics like academic behaviors, commitment, self-management and social support. The SuccessNavigator™ assessment not only provides scores and information, but also tailored feedback and action plans that can help both students, and those that work with them, focus on key areas of need.

We will then talk about three primary uses for the SuccessNavigator™ assessment: structuring the interactions between students and advisors, improving course placement decisions and targeting and tailoring interventions for groups of students, either within a classroom or across an institution.

On-screen: [Student Success and Holistic Assessment]

Graduation rates are critically low at institutions of higher education. Both two- and four-year schools face this challenge.

These figures show results from the 2003/2004 Beginning Postsecondary Students Study conducted by the Department of Education's National Center for Education Statistics. This study followed more than 16,000 students for six years, from the time they enrolled in the 2003/2004 academic year until 2009. One of the strong features of this study is that it follows students themselves, rather than institutional records of retention and graduation. Thus, we can see that even when taking into account transfer and varying enrollment patterns, only 64 percent of those students who enrolled in four-year institutions and 35 percent of students who enrolled in two-year institutions had received any sort of degree or certificate as of 2009.

On-screen: [Degree Attainment for Beginning College Students: Starting in 2003-04 Academic Year followed through 2009

Students Starting at Two-Year Institutions (pie chart):

Students Starting at Two-Year Institutions (pie chart):

If we apply these numbers to the total number of students who enrolled that year, we can estimate more than 1.2 million students had enrolled in 2003/2004 and failed to receive any degree or certificate by 2009.

Certainly, there is debate about how we might explain these findings or why there are such differences by institutional type, but regardless of those debates these numbers are staggeringly and unacceptably high.

One area that has received a great deal of attention recently is the academic preparation and subsequent course placement of beginning students, particularly those in community colleges. Here are some staggering statistics about college preparation:

According to a recent study by Complete College America, 50 percent of students who enroll at community colleges and 20 percent of those at four-year institutions are deemed unprepared for college-level coursework, and are placed into developmental and remedial courses.

Unfortunately, this places students, to use Complete College America’s term, on a “bridge to nowhere,” with fewer than 1 in 4 of these students receiving a degree.

On-screen: [More than 1.7 million students began at a community college in fall 2013 … But are they prepared?

Additionally, this may have a significant impact on the way students access and enroll in higher education. According to a 2008 study conducted by the Community College Research Center at Columbia University, 30 percent of students who are placed into developmental education never enroll at their institution. One explanation for this phenomenon is the discouraging effect of hearing that you are not ready for college.

There has been a litany of recommendations for addressing developmental education, including better alignment between secondary and postsecondary curricula, changes to course structure and expanded criteria for course placement. Regardless, these data show that how we place students in early courses is a critical decision point with long-term consequences for their success.

Although the paths to student success, and conversely the sources of attrition, have been modeled and studied a great deal over the past several decades, there are three key areas on which I’d like to focus today.

First, as we discussed, course placement has been recently cited as a critical juncture in student success. Students who are placed into remedial courses succeed at a drastically lower rate, and we in higher education must determine how we can better facilitate success for this population. However, course placement is just one issue in the larger picture of student success.

On-screen: [Where we lose students …
Course Placement: Students face a long, complex sequence of developmental courses
Early Academic Success: Students lack the effective behaviors (organization, study skills) that are needed to succeed in college-level courses
Persistence Over Time: Without well-developed and aligned goals, self-management skills and social connections, students may fail to persist to a degree]

For all students — regardless of where they are placed in the course spectrum — early academic success is critical. Here we can think of GPA as the focal outcome. Not only must students have the appropriate academic achievement when they enroll, but they must also have factors such as organization and time management to appropriately manage their coursework, in some cases balancing academic demands with a job, family, or other external pressures.

Moreover, we must consider students’ persistence to a degree, certificate, or other academic goal. We will discuss research that has shown that the factors that predict persistence behavior are different from those that predict academic success. Undoubtedly, classroom success and persistence are intertwined — students who earn good grades are more likely to come back each semester — but there are key differences in these phenomena, and we must appropriately understand them if we hope to intervene and improve rates of success.

Indeed, institutions of higher education are largely aware of these issues and the challenges that students face and frequently make efforts to address these areas. However, in our research and conversations with colleges and universities, we’ve identified several challenges that institutions face.

With regard to course placement, many institutions are seeking to shorten the developmental sequence by accelerating those students who are likely to succeed in higher-level courses. This, however, requires information in addition to traditional placement tests, which only measure one factor — academic achievement — of the many that determine student success.

Almost every college and university provides some sort of advising service to their students, designed to help them manage the curriculum, navigate obstacles and persist to a degree. However, these advisors are often challenged to identify which students have a low probability of success and thus need the highest levels of engagement. Moreover, they could benefit from targeted feedback and action plans across the spectrum of likely success.

Finally, from a macro-perspective, institutions invest heavily in the development of cocurricular programs and services to help benefit their students. However, what is “best practice” for one institution may not apply to the unique population of another school. Thus, colleges and universities need data, not only on the likely success of their students, but on how to provide and structure key programs and services that can facilitate persistence to a degree.

On-screen: [Challenges to Supporting Student Success
Course Placement:
Problem: High remediation rates lead to the "Bridge to Nowhere" (CCA, 2012)
Need: Identify students who might succeed on an accelerated path.

Advising:
Problem: Traditional indicators of student success are insufficient
Need: Better understanding of students' likelihood of success tied to course of action

Institutional Planning:
Problem: A host of programs and services based on "best practices"
Need: Data to target and structure programs and services to the population-specific strengths and weaknesses]

We propose that the best way to address each of these issues is through the use of a holistic assessment solution. Holistic assessment targets students early in their careers, facilitating interaction, setting a course for success and in many cases providing help for students before they even know they need it.

The term “holistic” refers to the consideration of both cognitive factors, that is academic achievement measured by indicators such as high school GPA and standardized placement or achievement test scores, and noncognitive or psychosocial factors. Research has shown that things like time management, motivation, self-efficacy and help-seeking behavior play a large role in student success, yet many institutions have little or no way of understanding these factors in their students.

Consider students A and B in this figure. If we were trying to place these students into an early math course based solely on their standardized placement test, student A would receive a placement score below our criterion for placement. However, this test failed to indicate that student A has strong time management skills, and can handle a more difficult course load. Moreover, student A frequently reaches out for help when facing a challenge. Using a holistic assessment solution, we might have understood that student A was actually quite likely to succeed. Conversely, student B scored well on the placement test, yet we failed to identify this student’s uncertainty about their degree plans. Moreover, student B often has high test anxiety, hindering performance on key exams. Although student B meets our criterion for success, there are still key challenges to face along the path to a degree, and a holistic assessment solution can help us identify those challenges and intervene before they hinder student B’s success.

On-screen: [A Holistic Assessment Solution: Bar charts for Student A and Student B with solid line for "criterion for success".
Student A: First bar shows "Placement Test Scores Alone" are insufficient to reach the criterion for success. Second bar shows Holistic Assessment (SuccessNavigator™) does allow the student to reach the criterion for success.

Student B: First bar shows "Placement Test Scores Alone" allow student to reach the criterion for success. Second bar shows Holistic Assessment (SuccessNavigator™) also allows the student to reach the criterion for success.]

And it’s actually this tie between the key student characteristics and institutional action that is the final and perhaps most important characteristic of a holistic assessment solution. It doesn’t do much good to understand the challenges a student faces if there is no subsequent course of action to mediate those risks. A holistic assessment solution seeks to turn data into information and information into action.

Let’s talk briefly about these psychosocial skills we’ve been talking about. This really refers to a large umbrella of factors that contribute to or are part of student learning. These skills generally have low correlations with academic achievement, but have been shown to play a large part in student success. This includes factors such as time management, study skills, commitment, motivation, test anxiety, self-efficacy and help-seeking behaviors.

On-screen: [What are psychosocial skills?
Social support, conscientiousness, metacognition, study skills, motivation, test-taking strategies, goal setting, institutional commitment, self-efficacy, teamwork.

Center of screen: Factors outside of academic achievement that contribute to or are part of student learning.]

Research has shown that psychosocial skills are indeed significant predictors of student success. A meta-analysis conducted by Steve Robbins and colleagues in 2004 looked at the relationship between academic, psychosocial and socioeconomic factors and student success. In predicting first-year GPA, standardized test scores had a sizable relationship with student grades, while socioeconomic status had a rather small effect. With regard to psychosocial factors, academic self-efficacy had an effect size similar to that of test scores, while other study and social factors had smaller effects.

On-screen: [Robbins, et aI. (2004) meta-analytic correlations with retention, GPA

Construct rGPA Retention
Academic Skills .129 .298
Academic Goals .155 .210
Academic Self-Efficacy .378 .257
Institutional Commitment .108 .204
Social Support .096 .199
SES .155 .173
ACT®/SAT® Scores .376 .079

Interestingly, when student retention was considered as an outcome, however, the relative importance of cognitive and noncognitive factors shifts. Factors like study skills, goals and social connections become sizably more important, while the importance of standardized test scores falls to the bottom of this list.

Over the past decade, this study, and others like it, have shown three key things when it comes to psychosocial or noncogntive skills:

First, they significantly predict success. Second, this effect is significant, even when controlling for factors like academic achievement and socioeconomic status. And third, the relative importance of these factors varies depending upon the outcome of interest — in this case, grades versus persistence.

On-Screen: [ETS® SuccessNavigator™. We're here to help. For additional guidance, please call 1-800-745-0269 or visit successnavigator.org.

[END]