By: Hans Sandberg
ETS has a long history of developing English-language assessments, such as TOEFL® and TOEIC® tests which are given internationally, and tests given to K–12 students in the United States. Focus on ETS R&D sat down with Larry Davis and Mikyung Wolf to learn more about ETS's research on assessing English-language proficiency.
What are the key challenges to today's tests of English-language proficiency?
Larry Davis: It is not easy to learn and become proficient in a language. It is a complex process, and this should be reflected in an assessment. Traditional, paper-based multiple-choice tests have provided a relatively easy means to measure a test taker's knowledge of grammar and vocabulary, but they have limitations when it comes to measuring a person's communicative-language skills — that is, measuring how well and appropriately a person can communicate with others in a given context.
Mikyung Wolf: We want tests to elicit adequate evidence of the test takers' abilities and provide better support for decision makers who want to know whether a person can communicate effectively in a real-world situation. But creating tests that measure communicative skills can be quite challenging when it comes to designing the test items, delivering and administrating the test, and collecting and scoring the test responses.
Larry Davis: Another challenge has to do with context. It can be hard for a test taker to respond to a question in a task if it is presented out of context. In a speaking task, the test taker may be asked to "tell us about a teacher who had a major impact in your life." But who is the test taker supposed to talk to? Also, without adequate context the test may not give us enough information about whether or not the test taker can communicate in a way that is appropriate, since that depends on the context.
Mikyung Wolf: ETS researchers are trying to find out if the test takers' responses vary if the context and the audience are explicitly described in the test items. Our researchers examined whether adding context to speaking items — of the type commonly found in large-scale tests — could provide more evidence about a test taker's ability to communicate effectively. One example is an argumentation task where test takers are asked to talk to someone and express their disagreement with an opinion. Will they do that in a polite way? Will they, for example, begin by trying to establish a rapport with the other person? Will they use the pronoun "you," as in "you said this," or "you said that?" A good speaker will consider who they are talking to and in which situation. With more context it becomes clearer how well the test takers communicate in real-life situations.
What do you do if you don’t get an answer, or get just a word or two? Can you help test takers show what they can do, rather than what they can't do?
Larry Davis: When a test taker doesn't respond, you'd like to know more, both from a decision-making and teaching standpoint. If a student is asked a question, but does not answer, then the teacher may ask a follow-up question to find out what the student actually knows. We've looked at doing that through a well-known pedagogical technique called scaffolding. The idea is that you pose a question and provide additional context in terms of information or key vocabulary words, if the student can’t answer. The student can also be asked a follow-up question that concentrates on just one part of the task.
Mikyung Wolf: Scaffolding can be compared to when you can "call a friend" in a quiz show. Recently, there is a growing realization that scaffolding can be an important feature of assessment. English-language assessments that include scaffolding could be particularly useful in classroom-based assessments, or assessments for instructional purposes, since the results could include the types of support or scaffolding the test takers needed, which gives more specific information about the test taker. In our research, we have explored ways of using scaffolding in standardized assessments. One challenge here is how to develop a score report when the test taker received help in the form of scaffolding. One solution could be to give partial credit if scaffolding was needed. A correct answer on the first try would earn full credit, while the credit would be reduced if the answer followed an intervention. The types and degree of scaffolding that the test taker used can also be included in a score report.
Larry Davis: One could also have a system that asks follow-up questions if the first answer is wrong, rather than incomplete. This could help the student answer and provide qualitative information about what the student knows or doesn't know.
Can new technology be used to get a better picture of the test takers' abilities?
Larry Davis: Yes, I think so. In fact, one of our main research areas is to investigate how to use technology to improve the features of English-language assessments. Increasingly, we are looking at ways to assess participation in group discussions, since interactive language skills are important for academic and workplace purposes.
When people ask how well somebody speaks English, they usually have in mind the ability to have a conversation with another person, and not just talking into a microphone. As technology improves, it may be feasible to assess people via Skype or similar technologies, where test takers speak "face-to-face" with an examiner or other test takers. This approach could make it possible to get better evidence of the ability of the test taker to interact with others, while still having the convenience of delivering the test via computer. Test takers don't need to be in the same room to talk with each other, so the logistics of doing the assessment becomes easier, and the computer platform can be used to deliver different kinds of content to talk about, or collaborative tasks to do. Besides, this approach also reflects how more and more people communicate today.
Mikyung Wolf: But new technology comes with its own challenges. For example, we may be able to embed multimedia to provide authentic inputs for language assessments. Yet, the internet speed and bandwidth to deliver the multimedia during large-scale testing may be problematic, depending on where the test is administered. Producing such technology-enhanced tasks can be costly as well. Also, we want to make sure that test takers are not held back by their computer skills when demonstrating their knowledge and skills in technology-enhanced language assessments. It is therefore important to think carefully about both theoretical and practical issues when using technology to assess language skills.
There is a lot of talk about using computer games or game-like components in learning. Could this improve assessment and make test takers more motivated?
Larry Davis: Motivation is crucial if we are going to get the best information about the test takers’ skills. If the test takers are not motivated, then we cannot be sure that they are fully engaged and are giving their best during the assessment. We’ve also explored ways to stimulate learner engagement and provide feedback by adding game-like features to tasks — but we are only in the beginning of this research.
Mikyung Wolf: Computer games can give students a goal to go after and reward them when they succeed. They can also get status updates about their performance. These features can be embedded in the assessment. Research suggests that such game-like features can improve motivation and learning, and we see this as a promising area — especially for classroom-based assessments, where assessment and instruction intersect closely.
Larry Davis: Context, scaffolding and new computer–based technologies are just a few examples of areas our researchers are working on in order to improve our understanding of English-language assessment. Such research can lead to new approaches and technologies that better serve the needs of English learners around the world, as well as in the United States.
Larry Davis is a Managing Research Scientist in ETS's R&D division, and Mikyung Wolf is a Managing Sr. Research Scientist in ETS's R&D division.
Language learning assessments
- English Language Learning and Assessment (ELLA)
- English Language Learning and Assessment Worldwide
- English Language Learning and Assessment in U.S. K–12 Education
Recent ETS Research Reports on English language assessment
- Defining and Operationalizing the Construct of Pragmatic Competence: Review and Recommendations (ETS Research Report No. RR-15-06)
- Creating a Next-Generation System of K–12 English Learner (EL) Language Proficiency Assessments (ETS Research Report No. RR-16-06)
- Key Issues and Opportunities in the Initial Identification and Classification of English Learners (ETS Research Report No. RR-16-09)
- Bootstrapping Development of a Cloud-Based Spoken Dialog System in the Educational Domain From Scratch Using Crowdsourced Data (ETS Research Report No. RR-16-16)