In response to the needs of the modern workplace, both education and assessment are placing an increasing emphasis on skills such as collaboration, critical thinking and technical literacy. Simultaneously, the use of computers and tablets for test delivery is enabling new task formats and assessment paradigms.
In order to leverage these exciting opportunities in ways that advance quality and equity in education, our researchers collaborate across disciplines, drawing upon expertise in areas that include, but are not limited to, test development, scoring technology, the cognitive and learning sciences, and advanced psychometrics. Our cross-disciplinary teams are interested in fundamental questions such as these:
- How do we define — in scientifically accepted ways — the skills to be measured?
- What types of interactive tasks allow students to best demonstrate their knowledge?
- Can the growing capacity for data collection, made possible by technology-enabled assessment, allow for more efficient measurement of complex skills?
- What kind of psychometric methods are necessary to analyze and enable valid use of these data?
The psychometric research behind next-generation assessments is crucial. Admissions officers, school administrators, prospective employers and others use test scores to make decisions that affect peoples' lives, and so it is important that the methods used to generate scores on the assessments of tomorrow be as sound as those used today and in the past.
Our research scientists and psychometricians have been particularly active in investigating the psychometric issues surrounding measurement of collaborative problem solving tasks, as well the use of games and simulations, and multimodal approaches to measure complex constructs that, to date, have not been well captured by educational assessments.
Collaborative Problem Solving
Next-generation assessments may be able to measure collaborative problem-solving skills, where several test takers work together to solve a problem. In order to realize the potential of such assessment capabilities, our cross-disciplinary teams are asking research questions such as these:
- What is the difference between students' performance on individual assessments and their performance on a corresponding collaborative assessment? What factors affect this difference, including characteristics of the team members, their interactions and the assessment task?
- What individual characteristics affect team performance? Is it possible to predict whether a team will perform well together?
- If individuals engage in multiple collaborations simultaneously, how might this affect the performance of different teams?
To address such issues, ETS researchers are exploring statistical models from various domains, such as social networks (maps of how individuals interact within a group) and time series (sequences of actions or events ordered by time). We are also seeking to facilitate interdisciplinary research on collaborative interactions and team work, working with researchers from the fields of organizational studies, healthcare and computer-based gaming.
Process Data from Educational Simulations and Games
Educational simulations and games could — if designed effectively — enable assessment of complex human behaviors such as problem solving and social negotiation. The key to taking advantage of such technology-enhanced assessments is the rich data they provide about a student's activity patterns within a task, rather than only reporting the final result.
More research is needed, however, to enable accurate and fair interpretations of the resulting complex process data. ETS research teams are investigating the significant challenges associated with these assessments, including data management and behavior-pattern recognition as well as psychometric modeling of interdependent, dynamic data (see "Psychometric Considerations in Game-Based Assessment" by Mislevy et al.). Our research scientists and psychometricians are exploring the potential of methods developed in other data-rich fields such as data mining, social networking and dynamic process modeling in order to facilitate this work.
Multimodal Analytics and Natural Interfaces
Just as humans simultaneously tap multiple senses to learn and gather information about their surroundings, multimodal analytics use advanced sensor technologies and machine learning to integrate information from multiple types of data streams, such as audio, video and keystroke data.
Our cross-disciplinary teams hope that these techniques will lead to assessments that measure human performance based on rich data captured automatically in environments designed to simulate natural settings. Multimodal analysis can also provide evidence of students' emotional states, such as boredom and frustration. Our researchers are interested in whether it is possible to use such clues to adapt assessments in order to help students perform their best and thus improve the accuracy of the measurement. To achieve these goals we are exploring new forms of assessments that involve automated analysis of speech and video to track verbal and non-verbal behaviors.